I do have a array which will read text file and do api call and then insert these result to mySQL. but when it run in bulk the response from API server is slow and due to this many of the results are coming blank. what I am looking is is there a way to pause this loop for each call say 5 seconds to get result from api server so it wont get blank results.
this code is below
//connect to your database
mysql_connect("localhost", "root", "password");
mysql_select_db("somedb");
//read your text file into an array, one entry per line
$lines = file('filename.txt');
//loop through each website URL
foreach ($lines as $website_url) {
//make the request to the compete API
$response = file_get_contents("http://apps.compete.com/sites/" . $website_url . "/trended/rank/?apikey=0sdf456sdf12sdf1");
//decode the request
$response = json_decode($request);
//get the rank from the response
$rank = $response['something'];
//insert the website URL and its rank
mysql_query("INSERT INTO website_ranks (website_url, rank) VALUES ('" . mysql_real_escape_string($website_Url) . "', " . $rank . ")");
}
Use the sleep command
sleep (5);
Wouldn't it make more sense to verify the server responded rather than sleeping for an arbitrary amount of time?
One alternate approach I'd recommend using instead of looping with a single INSERT statement is using a BULK INSERT syntax with your RDBMS provider. This will speed up the process considerably.
That syntax usually looks like this:
INSERT INTO tablename(col1, col2) VALUES (...)
Related
I have a PHP script on webserver 1 located in country A. I have a DB server located in country B.
The php script queries a large table from the DB server, groups the results, and inserts them back into the DB server (to another table). This is done by a single query (INSERT INTO SELECT...)
My question here is, does the data actually transfer between the web/db server? E.g. is this using GB's of bandwidth on both servers?
If you never deal with any retrieved data from DB server, then the query won't send any data to web server 1. Basically, if you just run the query, the only data that's sent is the text of the query (e.g. INSERT INTO SELECT...), which is probably just a few bytes, and then the response, which is just a success/fail value. That's it.
To say it another way, the data from the SELECT part of your INSERT INTO SELECT query is all dealt with on the DB server, it's never sent to webserver 1.
Even if you did run a SELECT query on a remote databse, you wouldn't get all of the results back. You actually get a resource. This resource is a reference to a set of data still on the remote database. When you do something like fetch_row on that resource, it fetches the next row. At that point, the data is transferred.
You can test this by monitoring the memory usage of your PHP script at various points in it's execution using memory_get_usage. Try:
echo "Memory before query: " . memory_get_usage() . "\n";
$result = $mysqli->query("your select query");
echo "Memory after query: " . memory_get_usage() . "\n";
$data = array();
$i=1;
while ($row = $result->fetch_row()) {
$data[] = $row;
echo "Memory after reading row " . $i++ . ": " . memory_get_usage() . "\n";
}
You should see a very small increase of used memory after your SELECT, and then a steady increase as your iterate over the results.
Hi I need to import a csv file of 15000 lines.
I m using the fgetcsv function and parsing each and every line..
But I get a timeout error everytime.
The process is too slow and data is oly partially imported.
Is there any way out to make the data import faster and more efficient?
if(isset($_POST['submit']))
{
$fname = $_FILES['sel_file']['name'];
$var = 'Invalid File';
$chk_ext = explode(".",$fname);
if(strtolower($chk_ext[1]) == "csv")
{
$filename = $_FILES['sel_file']['tmp_name'];
$handle = fopen($filename, "r");
$res = mysql_query("SELECT * FROM vpireport");
$rows = mysql_num_rows($res);
if($rows>=0)
{
mysql_query("DELETE FROM vpireport") or die(mysql_error());
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++)
{
if($i==1)
continue;
$sql = "INSERT into vpireport
(item_code,
company_id,
purchase,
purchase_value)
values
(".$data[0].",
".$data[1].",
".$data[2].",
".$data[3].")";
//echo "$sql";
mysql_query($sql) or die(mysql_error());
}
}
fclose($handle);
?>
<script language="javascript">
alert("Successfully Imported!");
</script>
<?
}
The problem is everytime it gets stuck in between the import process and displays the following errors:
Error 1 :
Fatal Error: Maximum time limit of 30 seconds exceeded at line 175.
Error 2 :
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'S',0,0)' at line 1
This error I m not able to detect...
The file is imported oly partial everytime.. oly around 200 300 lines out of a 10000 lines..
You can build a batch update string for every 500 lines of csv and then execute it at once if you are doing the mysql inserts on each line. It'll be faster.
Another solution is to read the file with an offset:
Read the first 500 lines,
Insert them to the database
Redirect to csvimporter.php?offset=500
Return the 1. step and read the 500 lines starting with offset 500 this time.
Another solution would be setting the timeout limit to 0 with:
set_time_limit(0);
Set this at the top of the page:
set_time_limit ( 0 )
It will make the page run endlessly. However, that is not recommended but if you have no other option then cant help!
You can consult the documentation here.
To make it faster, you need to check your the various SQL you are sending and see if you have proper indexes created.
If you are calling user defined functions and these functions are referring to global variables, then you can minimize the time take even more by passing those variables to the function and change the code so that the function refers to those passed variables. Referring to global variables is slower than local variables.
You can make use of LOAD DATA INFILE which is a mysql utility, this is much faster than fgetcsv
more information is available on
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
simply use this # the beginning of your php import page
ini_set('max_execution_time',0);
PROBLEM:
There is a huge performance impact on the way you INSERT data into your table. For every one of your records you send an INSERT request to the server, 15000 INSERT requests that's huge!
SOLUTION::
Well you should group your data like the way mysqldump does. In your case you just need three insert statement not 15000 as below:
before the loop write:
$q = "INSERT into vpireport(item_code,company_id,purchase,purchase_value)values";
And inside the loop concatenate the records to the query as below:
$q .= "($data[0],$data[1],$data[2],$data[3]),";
Inside the loop check that the counter is equal to 5000 OR 10000 OR 15000 then insert data to the vpireprot table and then set the $q to INSERT INTO... again.
run the query and enjoy!!!
If this is a one-time exercise, PHPMyAdmin supports Import via CSV.
import-a-csv-file-to-mysql-via-phpmyadmin
He also notes the user of leveraging MySQL's LOAD DATA LOCAL INFILE. This is a very fast way to import data into a database table. load-data Mysql Docs link
EDIT:
Here is some pseudo-code:
// perform the file upload
$absolute_file_location = upload_file();
// connect to your MySQL database as you would normally
your_mysql_connection();
// execute the query
$query = "LOAD DATA LOCAL INFILE '" . $absolute_file_location .
"' INTO TABLE `table_name`
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(column1, column2, column3, etc)";
$result = mysql_query($query);
Obviously, you need to ensure good SQL practices to prevent injection, etc.
I have a database with lots of game objects,
which is being queried by the following 3 PHP scripts.
List objects: gets a JSON object with all the items I need
Add object: adds an object to the database
Reset: wipes all objects from the table
All three of them work, somewhat. Although, there is a timing mismatch. When the game calls the reset function, it restarts. When the game restarts, it automatically loads all the objects. Unfortunately,and here's the problem, if the game has just been reset, objects will still be pulled by script 1.
I know of transactions, but I have never used them and I have no idea how I would implement those here, since my transaction is involving things from different scripts that are run at different times.
For bonus credit: will this setup (AS3 > php > MySQL) get me in trouble with a heavy load? The game might get picked up by 10, 100, 1000 people, is there anything I can read about that subject?
Edit: new idea/question
Currently, the wiping works as such: The objects table has a field 'deleted' which is set to '1' when the reset method is called. It might be smarter to copy the existing data into an archive table and then truncate the live table...
Edit: Here's the (relevant) PHP code I'm using
Add Object:
if ($db_found) {
$x = $_GET['x'];
$y = $_GET['y'];
$type = $_GET['type'];
$name = $_GET['name'];
$text = $_GET['text'];
$SQL = "INSERT INTO bodies (x,y,type,name,text)
VALUES ('".$x."','".$y."','".$type."','".$name."','".$text."' )";
if (!mysql_query($SQL))
{
die('Error: ' . mysql_error());
}
};
mysql_close($db_handle);
List/Get Objects:
if ($db_found) {
$SQL = "SELECT * FROM bodies WHERE deleted = 0";
$res = mysql_query($SQL);
$rows = array();
while($r = mysql_fetch_assoc($res)) {
print $r['x'] . ','
. $r['y']
. ','
. $r['type']
. ','
. $r['name']
. ','
. $r['text']
. ';';
}
};
mysql_close($db_handle);
Reset: (EDIT 2)
mysql_query("LOCK TABLES bodies WRITE;");
$SQL = " DELETE FROM bodies";
if (!mysql_query($SQL))
{
die('Error: ' . mysql_error());
}
};
mysql_query("UNLOCK TABLES;");
How to do Transactions in MySQL.
In your case you might be interessted in the atomicity and isolation of transactions, meaning that when restarting a game, you want to ensure that before the reset has not fully finished, nobody can fetch any of your intermediate data. Doing the reset inside a transaction will ensure this property*. (* for TRUNCATE see below)
You will need InnoDB as your Engine for all tables that are involved in your transactions. MyISAM does not support transactions.
Changing large amounts of data inside a transaction can potentially cause high query delays, as transaction use special undo/redo-logs to be able to undo all the things you did in your transaction, if you decide to ROLLBACK.
I wouldn't wipe the tables when starting a new game. Instead give your data a game_id and use a new game_id when starting a new game. Space shouldn't really be an issue nowadays. This has the advantage that you will need little to none table locking when reseting the game.
If you must, be sure to use TRUNCATE when clearing out the tables. As far as I know TRUNCATE in MySQL cannot be rolled back, so doing it inside a transaction won't do anything useful.
I think PHP/MySQL will perform fine if used correctly, even for larger visitor counts. You can use profiling tools like xdebug or the MySQL slow query log to trace and remove performance bottle necks.
I have an 800mb text file with 18,990,870 lines in it (each line is a record) that I need to pick out certain records, and if there is a match write them into a database.
It is taking an age to work through them, so I wondered if there was a way to do it any quicker?
My PHP is reading a line at a time as follows:
$fp2 = fopen('download/pricing20100714/application_price','r');
if (!$fp2) {echo 'ERROR: Unable to open file.'; exit;}
while (!feof($fp2)) {
$line = stream_get_line($fp2,128,$eoldelimiter); //use 2048 if very long lines
if ($line[0] === '#') continue; //Skip lines that start with #
$field = explode ($delimiter, $line);
list($export_date, $application_id, $retail_price, $currency_code, $storefront_id ) = explode($delimiter, $line);
if ($currency_code == 'USD' and $storefront_id == '143441'){
// does application_id exist?
$application_id = mysql_real_escape_string($application_id);
$query = "SELECT * FROM jos_mt_links WHERE link_id='$application_id';";
$res = mysql_query($query);
if (mysql_num_rows($res) > 0 ) {
echo $application_id . "application id has price of " . $retail_price . "with currency of " . $currency_code. "\n";
} // end if exists in SQL
} else
{
// no, application_id doesn't exist
} // end check for currency and storefront
} // end while statement
fclose($fp2);
At a guess, the performance issue is because it issues a query for each application_id with USD and your storefront.
If space and IO aren't an issue, you might just blindly write all 19M records into a new staging DB table, add indices and then do the matching with a filter?
Don't try to invent the wheel, it's been done. Use a database to search through the file's content. You can looad that file into a staging table in your database and query your data using indexes for fast access if they add value. Most if not all databases have import/loading tools to get a file into the database relatively fast.
19M rows on DB will slow it down if DB was not designed properly. You can still use text files, if it is partitioned properly. Recreating multiple smaller files, based on certain parameters, storing in proper sorted way might work.
Anyway PHP is not the best language for file IO and processing, it is much slower than Java for this task, while plain old C would be one of the fastest for the job. PHP should be restricted to generated dynamic Web output, while core processing should be in Java/C. Ideally it should be Java/C service which generates output, and PHP using that feed to generate HTML output.
You are parsing the input line twice by doing two explodes in a row. I would start by removing the first line:
$field = explode ($delimiter, $line);
list($export_date, ...., $storefront_id ) = explode($delimiter, $line);
Also, if you are only using the query to test for a match based on your condition, don't use SELECT * use something like this:
"SELECT 1 FROM jos_mt_links WHERE link_id='$application_id';"
You could also, as Brandon Horsley suggested, buffer a set of application_id values in an array and modify your select statement to use the IN clause thereby reducing the number of queries you are performing.
Have you tried profiling the code to see where it's spending most of its time? That should always be your first step when trying to diagnose performance problems.
Preprocess with sed and/or awk ?
Databases are built and designed to cope with large amounts of data, PHP isn't. You need to re-evaluate how you are storing the data.
I would dump all the records into a database, then delete the records you don't need. Once you have done that, you can copy those records wherever you want.
As others have mentioned, the expense is likely in your database query. It might be faster to load a batch of records from the file (instead of one at a time) and perform one query to check multiple records.
For example, load 1000 records that match the USD currency and storefront at a time into an array and execute a query like:
'select link_id from jos_mt_links where link_id in (' . implode(',', application_id_array) . ')'
This will return a list of those records that are in the database. Alternatively, you could change the sql to be not in to get a list of those records that are not in the database.
I have a thousands of data parsed from huge XML to be inserted into database table using PHP and MySQL. My Problem is it takes too long to insert all the data into table. Is there a way that my data are split into smaller group so that the process of insertion is by group? How can set up a script that will process the data by 100 for example? Here's my code:
foreach($itemList as $key => $item){
$download_records = new DownloadRecords();
//check first if the content exists
if(!$download_records->selectRecordsFromCondition("WHERE Guid=".$guid."")){
/* do an insert here */
} else {
/*do an update */
}
}
*note: $itemList is around 62,000 and still growing.
Using a for loop?
But the quickest option to load data into MySQL is to use the LOAD DATA INFILE command, you can create the file to load via PHP and then feed it to MySQL via a different process (or as a final step in the original process).
If you cannot use a file, use the following syntax:
insert into table(col1, col2) VALUES (val1,val2), (val3,val4), (val5, val6)
so you reduce to total amount of sentences to run.
EDIT: Given your snippet, it seems you can benefit from the INSERT ... ON DUPLICATE KEY UPDATE syntax of MySQL, letting the database do the work and reducing the amount of queries. This assumes your table has a primary key or unique index.
To hit the DB every 100 rows you can do something like (PLEASE REVIEW IT AND FIX IT TO YOUR ENVIRONMENT)
$insertOrUpdateStatement1 = "INSERT INTO table (col1, col2) VALUES ";
$insertOrUpdateStatement2 = "ON DUPLICATE KEY UPDATE ";
$counter = 0;
$queries = array();
foreach($itemList as $key => $item){
$val1 = escape($item->col1); //escape is a function that will make
//the input safe from SQL injection.
//Depends on how are you accessing the DB
$val2 = escape($item->col2);
$queries[] = $insertOrUpdateStatement1.
"('$val1','$val2')".$insertOrUpdateStatement2.
"col1 = '$val1', col2 = '$val2'";
$counter++;
if ($counter % 100 == 0) {
executeQueries($queries);
$queries = array();
$counter = 0;
}
}
And executeQueries would grab the array and send a single multiple query:
function executeQueries($queries) {
$data = "";
foreach ($queries as $query) {
$data.=$query.";\n";
}
executeQuery($data);
}
Yes, just do what you'd expect to do.
You should not try to do bulk insertion from a web application if you think you might hit a timeout etc. Instead drop the file somewhere and have a daemon or cron etc, pick it up and run a batch job (If running from cron, be sure that only one instance runs at once).
You should put it as said before in a temp directory with a cron job to process files, in order to avoid timeouts (or user loosing network).
Use only the web for uploads.
If you really want to import to DB on a web request you can either do a bulk insert or use at least a transaction which should be faster.
Then for limiting inserts by batches of 100 (commiting your trasnsaction if a counter is count%100==0) and repeat until all your rows were inserted.