I send a FULL flat (EDI) file to OXFORD everyday. I would query my database, get the array of results, and use the data array to make a csv file out of it
$sql = "SELECT * FROM master_table";
$sth = $apex->prepare($sql);
$sth->execute();
$result = $sth->fetchAll(PDO::FETCH_ASSOC); //this is where I would get my fatal error due to out of memory
$csv = new csv();
$csv->makeCsv($result);
The $result would contain an array of millions of records which I then make into a csv.
The main problem I'm having here is a lack of memory and time. However if I "break up" the sql query like this:
$years = [2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011]; //etc
foreach ($years as year) {
$sql = "SELECT * FROM master_table WHERE year = $year";
$sth = $apex->prepare($sql);
$sth->execute();
$result = $sth->fetchAll(PDO::FETCH_ASSOC);
$csv = new csv();
$csv->makeCsv($result);
unset($result);
}
This works but it takes a extremely long time due to php not being unable to execute the foreach loop simultaneously. Is there an option in PDO where I can execute multithreaded queries?
Even if you could select the data faster from multiple threads (which you can't most probably), you couldn't write the CSV any faster whatever ... threading doesn't make sense here.
The best solution is to SELECT INTO OUTFILE, bypassing PHP in the creation of the CSV altogether.
Related
In my database, I have huge tables with a couple of billions of results and the problems are that my PHP code is not capable of getting it right. I'm waiting for hours of results and almost always get an error like 'served moved away' or 'maximum limit of passing data is xx'.
Tables have indexed columns. Queries are simplified to the minimum. I have tried some test in PHP but all of them failed.
// approach #1 with PHP
$pdo = new PDO("mysql:host=".$data["host"].";dbname=".$data["dbname"].";charset=utf8", $data["username"], $data["pass"]);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$sql = "SELECT id FROM huge_table bt INNER JOIN even_bigger_table ebt ON bt.id = ebt.id WHERE bt.column = 'data1' AND ebt.column2 = 'data2'";
// ends up with error with no results
$pdo->query($sql)->fetchAll();
// approach #2 with PHP
$pdo = new PDO("mysql:host=".$data["host"].";dbname=".$data["dbname"].";charset=utf8", $data["username"], $data["pass"]);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$resultsCounted = 5000000;
$offset = 0;
for($i = 0; $offset < $resultsCounted; $i += 500) {
$sql = "SELECT id FROM huge_table bt INNER JOIN even_bigger_table ebt ON bt.id = ebt.id WHERE bt.column = 'data1' AND ebt.column2 = 'data2' LIMIT ".$offset.", 500";
// ends up with error with no results
$result = $pdo->query($sql)->fetchAll();
// do some stuff with results
do_stuff($result);
$offset += $i;
}
Strangely this SQL works in PhpStorm database console when selecting "Execute to file" function:
SELECT id FROM huge_table bt INNER JOIN even_bigger_table ebt ON bt.id = ebt.id WHERE bt.column = 'data1' AND ebt.column2 = 'data2'
I can see how PhpStorm saves 100 by 100 results into the file until all are saved.
I need to find a way how can I recreate PhpStorm database data saving that with PHP or find another way how to properly manage results saving.
LIMIT hugenumber smallnumber generally works poorly: it's a notorious performance antipattern. MySQL has to work hard to skip over a hugenumber of records before returning a smallnumber. Then it has to do it all again, and again...
Instead process your rows one at a time.
Most queries in php are buffered. That is, fetchAll() and its equivalents slurp up everything in the query resultset in one big gulp.
That works badly for queries with vast resultsets, partly because most php implementations have RAM limits.
Use an unbuffered query for this. Process the rows you need one at a time, then discard them. That way your resultset will flow through your php program in a continuous stream. (I did not debug this example).
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
$sql = "SELECT id FROM huge_table bt INNER JOIN even_bigger_table ebt ON bt.id = ebt.id WHERE bt.column = 'data1' AND ebt.column2 = 'data2'";
$unbuff = $pdo->query($sql);
if ($unbuff) {
while ($row = $unbuff->fetch(PDO::FETCH_ASSOC)) {
echo $row['whatever']; /* handle the row */
}
}
I want to run multiple sql queries in a single database connection using oracle 10g and php. Here for every sql query queries I have to create database connection. Is there any way to run multiple sql queries in a single database connection or we have to fetch data this way only? Because when we have to run 50 queries, we have to write 50 times like below.
<?php
include("mydb.php");
// run query
$sql6 = "select * from dat where to_char(WD,'dd/mm')='19/08'";
$stid6=oci_parse($conn, $sql6);
// set array
$arr6 = array();
if(!$stid6){
$e=oci_error($conn);
trigger_error(htmlentities($e[message],ENT_QUOTES),E_USER_ERROR);
}
$r6=oci_execute($stid6);
if(!$r6){
$e=oci_error($stid6);
trigger_error(htmlentities($e[message],ENT_QUOTES),E_USER_ERROR);
}
// look through query
while($row = oci_fetch_array($stid6,OCI_ASSOC)){
// add each row returned into an array
$arr6[] = array(($row['WD']) , (float)$row['DATA']);
}
oci_free_statement($stid6);
oci_close($conn);
?>
<?php
include("mydb.php");
// run query
$sql7 = "select * from dat where to_char(WD,'dd/mm')='11/03'";
$stid7 = oci_parse($conn, $sql7);
// set array
$arr7 = array();
if(!$stid7){
$e=oci_error($conn);
trigger_error(htmlentities($e[message],ENT_QUOTES),E_USER_ERROR);
}
$r7=oci_execute($stid7);
if(!$r7){
$e=oci_error($stid7);
trigger_error(htmlentities($e[message],ENT_QUOTES),E_USER_ERROR);
}
// look through query
while($row = oci_fetch_array($stid7,OCI_ASSOC)){
// add each row returned into an array
$arr7[] = array(($row['WD'])) , (float)$row['DATA']);
}
oci_free_statement($stid7);
oci_close($conn);
?>
................
................
*Pardon me, I forgot to mention that we have store day-wise data in different array. I mean to say that 11/03's data will store in arr1 and 19/08's data will be stored in arr2. Not in same array.
(this should be a comment, but its a bit long)
I don't want to be disparaging here, but your question is alarming naive - so much so that it should be closed as off-topic.
Your code exhibits a lack of understanding about modular programming and variable scope. These should be covered by day 2 of a programming-from-scratch course. but oddly includes some more sophisticated PHP specific programming, but apalling SQL - it looks like someone else wrote the code as a quick hack and now you are trying to extend its capabilities.
That you are using an Oracle database, raises all sorts of questions about why are attempting this (Oracle is expensive; how can someone afford that but not afford to provision you with the skills you need?)
The solution to the problem as you have described it is to re-implement the script as a function that takes the OCI connection and SQL statement as parameters, then simply....
<?php
include("mydb.php");
$queries=array(
"select * from dat where to_char(WD,'dd/mm')='11/03'",
"select * from dat where to_char(WD,'dd/mm')='19/08'"
);
foreach ($queries as $sql) {
run_qry($sql, $conn);
}
oci_close($conn);
exit;
function run_query($sql, $conn)
{
$stid=oci_parse($conn, $sql);
// set array
$arr = array();
if(!$stid){
$e=oci_error($conn);
trigger_error(htmlentities($e[message],ENT_QUOTES),E_USER_ERROR);
}
...
oci_free_statement($stid);
return $arr;
}
However since the 2 example queries have exactly the same structure, there are other ways to the results of what are multiple queries - merging the SQL states into a single select using OR or UNION, using parameterized queries. Since the code you've shown us simply throws away the results its hard to say how you should approach the task.
So here is my scenario...
The bug_tracker table is in one server and task_traker is in another.
I want to show a combined result but can't since there are in two separate databases remotely.
So I am calling the task tracker first and then getting the bug details per iteration.
$task = oci_parse($task_conn, "select * from task_table where ....");
oci_execute($task);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
$bug = oci_parse($bug_conn, "select * from bug_table where id = " . $task_row['BUGID'] );
oci_execute($bug);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
... //output
}
... //output
}
But this entire process is making it very slow... since there are large number of records and columns.
Is there any way to make it even slightly faster? Note: I don't have access so can't setup oracle db links.
You could improve it using the IN statement:
<?php
$task = oci_parse($task_conn, "select * from task_table where ....");
oci_execute($task);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
$bugs[] = $task_row['BUGID'];
$users[] = $task_row['USER'];
$status[] = $task_row['TASK_STATUS'];
}
$bug = oci_parse($bug_conn, "select * from bug_table where id IN (" . implode(',', $bugs) . ");" );
oci_execute($bug);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
// ...
}
?>
On a sidenote, why are you not using PDO? I believe using it will already give you a performance boost.
PHP is not meant for this kind of operation, neither should you try to write your own join function.
One proper way of solving this issue is to dump the data from both databases into a local database, and there do the join.
You do not need anything fancy for the local database, an SQLite3 is probably enough.
Just dump the data from each database into a CSV files using a bash script that you put into cron. After the dump, (re)create each table in your SQLite3, and load the CSVs into these tables. After this you can do a join once and push the result into a new table which you then are free to query.
This is what in the datawarehouse world is often referred to as an ETL process, just in this case, very very simplified.
I'm getting really tired of writing:
$rows = array();
$result = $db->query("SELECT * FROM `table` WHERE `field`='".$information."'");
while($row = $result){
$rows[] = $row;
}
I tried a function, but it was kinda of messy feeling changing the input to field inputs. I thought maybe this or something similar would help:
$rows = array();
while($row = ($db->query("SELECT * FROM `table` WHERE `field`='".$information."'"))->fetch_assoc()){
$rows[] = $row;
}
but I get unexpected T_OBJECT_OPERATOR. I'm still on the line about using a function. Maybe there's a more efficient way of writing one. This is how I tried writing a function:
$array = SELECT ($db,$toSelect,$table,$where);
It still seems cumbersome, however. I would like something like $array = $db->("MYSQL");
The simplest solution is to write a function, which expects a database handle and a string query parameter, and returns all the rows.
$rows = fetch_all($db, "SELECT ...");
A bit more advanced is to write your own database class which wraps the database handle and adds such functionality.
$rows = $mydb->fetch_all("SELECT ...");
If you don't want to reinvent the wheel, simply use an existing ORM / PHP database library which does all this (and more) for you.
$db
->select('*')
->from('table')
->where('field', $information);
Note: http://www.php.net/manual/en/security.database.sql-injection.php - the third solution automatically solves this problem.
I want to show all text messages from db where id=$e ($err is an array).
Inserted the query into the foreach loop, it works well but it does extra work (does query for every value of array).
Is there any other way to do it (i mean extract query from foreach loop)?
My code looks like this.
foreach ($err as $e)
{
$result = $db -> query("SELECT * from err_msgs WHERE id='$e'");
$row = $result -> fetch_array(MYSQLI_BOTH);
echo "<li><span>".$row[1]."</span></li>";
}
It is much more efficient to do this with implode() because it will only result in one database query.
if (!$result = $db->query("SELECT * FROM `err_msgs` WHERE `id`='".implode("' OR `id`='",$err)."'")) {
echo "Error during database query<br />\n";
// echo $db->error(); // Only uncomment this line in development environments. Don't show the error message to your users!
}
while ($row = $result->fetch_array(MYSQLI_BOTH)) {
echo "<li><span>".$row[1]."</span></li>\n";
}
Check the SQL IN clause.
Firstly, a bit of a lecture: embedding strings directly into your queries is going to cause you trouble at some point (SQL injection related trouble to be precise), try to avoid this if possible. Personally, I use the PDO PHP library which allows you to bind parameters instead of building up a string.
With regard to your question, I'm not sure I have understood. You say that it does extra work, do you mean that it returns the correct results but in an inefficient way? If so then this too can be addressed with PDO. Here's the idea.
Step 1: Prepare your statement, putting a placeholder where you currently have '$e'
Step 2: Loop through $err, in the body of the loop you will set the place holder to be the current value of $e
By doing this you not only address the SQL injection issue, you can potentially avoid the overhead of having to parse and optimise the query each time it is executed (although bear in mind that this may not be a significant overhead in your specific case).
Some actual code would look as follows:
// Assume that $dbdriver, $dbhost and $dbname have been initialised
// somewhere. For a mysql database, the value for $dbdriver should be
// "mysql".
$dsn = "$dbdriver:host=$dbhost;dbname=$dbname";
$dbh = new PDO($dsn, $dbuser, $dbpassword);
$qry = "SELECT * from err_msgs WHERE id = :e"
$sth = $dbh->prepare($qry);
foreach ($err as $e) {
$sth->bindParam(":e", $e);
$sth->execute();
$row = $sth->fetch();
// Prints out the *second* field of the record
// Note that $row is also an associative array so if we
// have a field called id, we could use $row["id"] to
// get its value
echo "<li><span>".$row[1]."</span></li>";
}
One final point, if you want to simply execute the query once, instead of executing it inside the loop, this is possible but again, may not yield any performance improvement. This could be achieved using the IN syntax. For example, if I'm interested in records with id in the set {5, 7, 21, 4, 76, 9}, I would do:
SELECT * from err_msgs WHERE id IN (5, 7, 21, 4, 76, 9)
I don't think there's a clean way to bind a list using PDO so you would use the loop to build up the string and then execute the query after the loop. Note that a query formulated in this way is unlikely to give you any noticable performance improvment but it really does depend on your specific circumstances and you'll just have to try it out.
You can do this much simpler by doing
$err_csv = implode("','",$err);
$sql = "SELECT FROM err_msgs WHERE id IN ('$err_csv')";
$result = $db -> query($sql);
while ($row = $result -> fetch_array(MYSQLI_BOTH))
{
echo "<li><span>".$row[1]."</span></li>";
}
That way you don't have to keep sending queries to the database.
Links:
http://php.net/manual/en/function.implode.php