What I'm trying to do is copy the result of a query in oracle into table in a MySQL database. The reasoning behind this is not important to the question, and I can't really take a different approach.
What I am doing is running the query though php, then copying the result into a newly created table in oracle. I know this is not very efficient, but my table sizes are pretty small, even though the queries run very long.
Everything works other than I'm having trouble bringing the date over from oracle; when I run it as is, my date fields are set to 0. What I'm doing is checking the result set from oracle to see if the type of a column is "DATE(7)" and if it is then I create a column in MySQL with type "DATE". But for some reason this doesn't work.
I'm using the following code, sorry it's quite long, but it seemed best to provide it all.
function hmis_query_transfer ($query_name, $parameters = NULL) {
//Create Connections to both servers
$dbConn_Oracle = hmis_oci_connect();
hmis_mysql_connect("hmis_temp");
//Retrieve the text for the query and run it on the Oracle server
$query_Oracle = hmis_query_by_name($query_name, $parameters);
$stmt_Oracle = hmis_oci_query($dbConn_Oracle , $query_Oracle);
$ncols = oci_num_fields($stmt_Oracle);
//Test if table is already created in MySQL
$mysql_check = "DESC ".$query_name;
#hmis_mysql_query($mysql_check);
if (mysql_errno()==1146){
$mysql_create = "CREATE TABLE ".$query_name." ( ";
//Transform and append column names to query string
for ($j = 1; $j <= $ncols; $j++) {
$name = oci_field_name($stmt_Oracle, $j);
$type = oci_field_type($stmt_Oracle, $j);
if ($type == "NUMBER") {
$type = "INT";
} else if ($type == "VARCHAR2"){
$type = "VARCHAR";
}
$type .= "(".oci_field_size($stmt_Oracle, $j).")";
if ($type == "DATE(7)") {
$type = "DATE";
}
$mysql_create .= $name." ".$type.",";
}
$mysql_create = substr_replace( $mysql_create, "", -1 );
$mysql_create .= ')';
//Create Table
$result = hmis_mysql_query($mysql_create);
if ( !$result ){
die('<strong>Failed Create:</strong>'.mysql_error());
}
}
elseif (!mysql_errno()) {
//If the table already exists, empty it
$mysql_truncate = "TRUNCATE TABLE ".$query_name;
$result = hmis_mysql_query($mysql_truncate);
if ( !$result ){
die('<strong>Failed Truncate:</strong>'.mysql_error());
}
}
//Copy over row by row the data from the result set to MySQL
while ($results_row = oci_fetch_array($stmt_Oracle, OCI_ASSOC )) {
$mysql_insert = "INSERT INTO ".$query_name." VALUES (";
for ($i = 1; $i <= $ncols; $i++) {
$mysql_insert .= "'".$results_row[oci_field_name($stmt_Oracle, $i)]."',";
}
$mysql_insert = substr_replace( $mysql_insert, "", -1 );
$mysql_insert .= ")";
$result = hmis_mysql_query($mysql_insert);
if ( !$result ){
die('<strong>Failed Insert:</strong>'.mysql_error());
}
}
}
Can anyone see any flaws in my code? I'm open to suggestions on how I could do this a different way, though I would prefer to be able to keep my code. Thanks for any help.
The reason I'm copying data from oracle to MySQL is because most of the queries take very long to run (15-20 min), and deal wit huge datasets (200 million), but my result sets are very small(a few thousand at most). The way my application was built it has to run repeat queries on the same dataset to accomplish it's task. I would have liked to create views to accomplish this in oracle, but I don't have the authority to do so. Therefore I save an mid layer result set and perform my analysis on it, which happens to be on a much faster machine.
I do not know the format of the date from Oracle, but I would use the strtotime() function to convert it to a unix timestamp and then use the date() function to put it into MySQL (YYYY-MM-DD) format. This should work as long as the date field in oracle is not suppose to be a birthdate or out of the range of a UNIX Timestamp.
Related
I have a MySQL table with the following fields:
ID
PHONE
NAME
CITY
COUNTRY
Using PHP, I am reading a comma separated dump of values off a text document, parsing the values and inserting records to the table. For reference, here's the code:
<?php
// Includes
require_once 'PROJdbconn.php';
// Read comma-separated text file
$arrindx = 0;
$i = 0;
$filehandle = fopen(PROJCDUMPPATH.PROJCDUMPNAME,"rb");
while (!feof($filehandle)){
$parts = explode(',', fgets($filehandle));
$contnames[$arrindx] = $parts['0'];
$contnumbers[$arrindx] = preg_replace('/[^0-9]/','',$parts['1']);
$arrindx += 1;
}
fclose($filehandle);
$arrindx -= 1;
$filehandle = NULL;
$parts = NULL;
// Build SQL query
$sql = "INSERT INTO Contact_table (PHONE, NAME) VALUES ";
for ($i = 0; $i < $arrindx; ++$i){
$sql .= "('".$contnumbers[$i]."', '".$contnames[$i]."'),";
}
$i = NULL;
$arrindx = NULL;
$contnames = NULL;
$contnumbers = NULL;
$sql = substr($sql,0,strlen($sql)-1).";";
// Connect to MySQL database
$connect = dbconn(PROJHOST,PROJDB,PROJDBUSER,PROJDBPWD);
// Execute SQL query
$query = $connect->query($sql);
$sql = NULL;
$query = NULL;
// Close connection to MySQL database
$connect = NULL;
?>
Now, this code, as you can see, blindly dumps all records into the table. However, I need to modify the code logic as such:
Read text file and parse records into arrays (already doing)
For each record in text file
Check if PHONE exists in the table
If yes,
For each field in the text file record
If text file field != NULL
Update corresponding field in table
Else
Skip
If no,
INSERT record (already doing)
I apologize if the logic isn't terribly clear, feel free to ask me if any aspect confuses you. So, I understand this logic would involve an insane number of SELECT, UPDATE, and INSERT queries, depending on the number of fields (I intend to add more fields in future) and records. Is there any way to either somehow morph them into a single query or leastwise optimize the code by minimizing the number of queries?
What you're trying to do is called an "upsert" (update/insert).
MySQL INSERT else if exists UPDATE
I am really trying to wrap my head around this and failing miserably. What I want to do it build a MySQL query based on the URL parameters passed by the URL. I am trying to create a re usable dynamic script that can do what it needs to do based on the URL parameter.
This is what I have come up with, and it appears that it does what it is supposed to do (no errors or anything) but nothing actually gets inserted in the database. I know somewhere I have made a dumb mistake (or thought something out wrong) so hopefully one of you guys can point me in the right direction.
Thanks!
//List all possible variables you can expect the script to receive.
$expectedVars = array('name', 'email', 'score', 'age', 'date');
// This is used for the second part of the query (WHERE, VALUES, ETC)
$fields = array('uName','uEmail','uScore','uAge','uDate');
// Make sure some fields are actually populated....
foreach ($expectedVars as $Var)
{
if (!empty($_GET[$Var]))
{
$fields[] = sprintf("'%s' = '%s'", $Var, mysql_real_escape_string($_GET[$Var]));
}
}
if (count($fields) > 0)
{
// Construct the WHERE Clause
$whereClause = "VALUES " . implode(",",$fields);
//Create the SQL query itself
$sql = ("INSERT INTO $mysql_table ($fields) . $whereClause ");
echo "1"; //It worked
mysql_close($con);
}
else
{
// Return 0 if query failed.
echo "0";
}
?>
You missed mysql_query($sql):
if(!mysql_query($sql)){
//die(mysql_error());
}
Please consider to use PDO or My SQLi using parametrize query because mysl_* function depreciated.
Your SQL is all wrong. You're using the field = value syntax for an INSERT, then you're concatenating an array as if it were a string ($fields), and you're missing a couple of parentheses around the values.
a couple of things: i've found for php <-> mysql its important to see what's going into mysql and experiement directly with those queries in phpmyadmin when i get stuck.
1 - in my code I output mysql_error() when the query fails or when a debug flag is set. this usually explains the sql issue in a way that can point me to a misspelled field name etc...
2 - this way i can feed that mysql query directly into phpmyadmin and tweak it until it gives me the results i want. (while i'm there i can also use explain to see if i need to optimize the table)
specifics in your code. unlike C languages sprintf is implied. here's how i'd write your code:
// List all possible variables you can expect the script to receive.
$expectedvars = array('name', 'email', 'score', 'age', 'date');
// This is used for the second part of the query (WHERE, VALUES, ETC)
// $fields = array('uName','uEmail','uScore','uAge','uDate');
$fields = array();
// Set only the variables that were populated ...
foreach ($expectedvars as $var) {
if (!empty($_GET[$var])) {
$name = "u" + ucwords($var); // convert var into mysql field names
$fields[] = "{$name} = " . mysql_real_escape_string($_GET[$var]);
}
}
// only set those fields which are passed in, let the rest use the mysql default
if (count($fields) > 0) {
// Create the SQL query itself
$sql = "INSERT INTO {$mysql_table} SET " . implode("," , $fields);
$ret = mysql_query($sql);
if (!$ret) {
var_dump('query_failed: ', $sql, $ret);
echo "0"; // Query failed
} else {
echo "1"; // It worked
}
} else {
// Return 0 if nothing to do
echo "0";
}
mysql_close($con);
Is there a way to filter the queries by table name in the zend db profiler? The documentation doesn't have anything but I don't know if I can just rely on this document completely..if you know a way, please advise..
There is currently not a way to filter the profiler based on table name, only by query type (INSERT, UPDATE, etc) or elapsed time of the query.
Here is some code you could try that may help you do what you want though, note, I have not tested it, but hopefully it can get you started.
Basically, it loops over each query that was profiled and uses preg_match to see if the query was to your table, and if not it unsets the query info and continues, if it was then it updates some stats. At the end of the foreach, $queries, should be only the queries to the table you want to profile.
<?php
$tableName = 'my_table';
/** var $profiler Zend_Db_Profiler */
$profiler = $db->getProfiler();
$queries = $profiler->getQueryProfiles();
$totalQueries = 0;
$totalTime = 0;
if ($queries !== false) {
foreach ($queries as $index => $query) {
$queryString = $query->getQuery();
$t = preg_quote($tableName);
if (!preg_match("/UPDATE .?$t.? /i", $queryString) ||
!preg_match("/INSERT INTO .?$t.?/i", $queryString) ||
!preg_match("/DELETE FROM .?$t.?/i", $queryString) ||
!preg_match("/REPLACE .*?INTO .?$t.?/i", $queryString) ||
) {
unset($queries[$index]);
continue;
}
$totalQueries++;
$totalTime += $query->getElapsedSecs();
}
}
Please bear with me on this question.
I'm looking to create a relatively large MySQL database that I want to use to do some performance testing. I'm using Ubuntu 11.04 by the way.
I want to create about 6 tables, each with about 50 million records. Each table will have about 10 columns. The data would just be random data.
However, I'm not sure how I can go about doing this. Do I use PHP and loop INSERT queries (bound to timeout)? Or if that is inefficient, is there a way I can do this via some command line utility or shell script?
I'd really appreciate some guidance.
Thanks in advance.
mysql_import is what you want. Check this for full information. It's command line and very fast.
Command-line mode usually has the timeouts disabled, as that's a protection against taking down a webserver, which doesn't apply at the command line.
You can do it from PHP, though generating "random" data will be costly. How random does this information have to be? You can easily read from /dev/random and get "garbage", but it's not a source of "good" randomness (You'd want /dev/urandom, then, but that will block if there isn't enough entropy available to make good garbage).
Just make sure that you have keys disabled on the tables, as keeping those up-to-date will be a major drag on your insert operations. You can add/enable the keys AFTER you've got your data set populated.
If you do want to go the php way, you could do something like this:
<?php
//Edit Following
$millionsOfRows = 2;
$InsertBatchSize = 1000;
$table = 'ATable';
$RandStrLength = 10;
$timeOut = 0; //set 0 for no timeout
$columns = array('col1','col2','etc');
//Mysql Settings
$username = "root";
$password = "";
$database = "ADatabase";
$server = "localhost";
//Don't edit below
$letters = range('a','z');
$rows = $millionsOfRows * 1000000;
$colCount = count($columns);
$valueArray = array();
$con = #mysql_connect($server, $username, $password) or die('Error accessing database: '.mysql_error());
#mysql_select_db($database) or die ('Couldn\'t connect to database: '.mysql_error());
set_time_limit($timeOut);
for ($i = 0;$i<$rows;$i++)
{
$values = array();
for ($k = 0; $k<$colCount;$k++)
$values[] = RandomString();
$valueArray[] = "('".implode("', '", $values)."')";
if ($i > 0 && ($i % $InsertBatchSize) == 0)
{
echo "--".$i/$InsertBatchSize."--";
$sql = "INSERT INTO `$table` (`".implode('`,`',$columns)."`) VALUES ".implode(',',$valueArray);
mysql_query($sql);
echo $sql."<BR/><BR/>";
$valueArray = array();
}
}
mysql_close($con);
function RandomString ()
{
global $RandStrLength, $letters;
$str = "";
for ($i = 0;$i<$RandStrLength;$i++)
$str .= $letters[rand(0,25)];
return $str;
}
Of course you could just use a created dataset, like the NorthWind Database.
all you need to do is launch your script from command line like this:
php -q generator.php
it can then be a simple php file like this:
<?php
$fid = fopen("query.sql", "w");
fputs($fid, "create table a (id int not null auto_increment primary key, b int, c, int);\n");
for ($i = 0; $i < 50000000; $i++){
fputs($fid, "insert into table a (b,c) values (" . rand(0,1000) . ", " . rand(0,1000) . ")\n");
}
fclose($fid);
exec("mysql -u$user -p$password $db < query.sql");
Probably it is fastest to run multiple inserts in one query as:
INSERT INTO `test` VALUES
(1,2,3,4,5,6,7,8,9,0),
(1,2,3,4,5,6,7,8,9,0),
.....
(1,2,3,4,5,6,7,8,9,0)
I created a PHP script to do this. First I tried to construct a query that will hold 1 million inserts but it failed. Then I tried with 100 thousend and it failed again. 50 thousends don't do it also. My nest try was with 10 000 and it works fine. I guess I am hitting the transfer limit from PHP to MySQL. Here is the code:
<?php
set_time_limit(0);
ini_set('memory_limit', -1);
define('NUM_INSERTS_IN_QUERY', 10000);
define('NUM_QUERIES', 100);
// build query
$time = microtime(true);
$queries = array();
for($i = 0; $i < NUM_QUERIES; $i++){
$queries[$i] = 'INSERT INTO `test` VALUES ';
for($j = 0; $j < NUM_INSERTS_IN_QUERY; $j++){
$queries[$i] .= '(1,2,3,4,5,6,7,8,9,0),';
}
$queries[$i] = rtrim($queries[$i], ',');
}
echo "Building query took " . (microtime(true) - $time) . " seconds\n";
mysql_connect('localhost', 'root', '') or die(mysql_error());
mysql_select_db('store') or die(mysql_error());
mysql_query('DELETE FROM `test`') or die(mysql_error());
// execute the query
$time = microtime(true);
for($i = 0; $i < NUM_QUERIES; $i++){
mysql_query($queries[$i]) or die(mysql_error());
// verify all rows inserted
if(mysql_affected_rows() != NUM_INSERTS_IN_QUERY){
echo "ERROR: on run $i not all rows inserted (" . mysql_affected_rows() . ")\n";
exit;
}
}
echo "Executing query took " . (microtime(true) - $time) . " seconds\n";
$result = mysql_query('SELECT count(*) FROM `test`') or die(mysql_error());
$row = mysql_fetch_row($result);
echo "Total number of rows in table: {$row[0]}\n";
echo "Total memory used in bytes: " . memory_get_usage() . "\n";
?>
The result on my Win 7 dev machine are:
Building query took 0.30241012573242 seconds
Executing query took 5.6592788696289 seconds
Total number of rows in table: 1000000
Total memory used in bytes: 22396560
So for 1 mil inserts it took 5 and a half seconds. Then I ran it with this settings:
define('NUM_INSERTS_IN_QUERY', 1);
define('NUM_QUERIES', 1000000);
which is basically doing one insert per query. The results are:
Building query took 1.6551470756531 seconds
Executing query took 77.895285844803 seconds
Total number of rows in table: 1000000
Total memory used in bytes: 140579784
Then I tried to create a file with one insert per query in it, as suggested by #jancha. My code is slightly modified:
$fid = fopen("query.sql", "w");
fputs($fid, "use store;");
for($i = 0; $i < 1000000; $i++){
fputs($fid, "insert into `test` values (1,2,3,4,5,6,7,8,9,0);\n");
}
fclose($fid);
$time = microtime(true);
exec("mysql -uroot < query.sql");
echo "Executing query took " . (microtime(true) - $time) . " seconds\n";
The result is:
Executing query took 79.207592964172 seconds
Same as executing the queries through PHP. So, probably the fastest way is to do multiple inserts in one query and shouldn't be a problem to use PHP to do the work.
Do I use PHP and loop INSERT queries (bound to timeout)
Certainly running long duration scripts via a webserver mediated requset is not a good idea. But PHP can be compiled to run from the command line - in fact most distributions of PHP come bundled with this.
There are lots of things you do to make this run more efficiently, exactly which ones will vary depedning on how you are populating the data set (e.g. once only, lots of batch additions). However for a single load, you might want to have a look at the output of mysqldump (note disabling, enabling indexes, multiple insert lines) and recreate this in PHP rather than connecting directly to the database from PHP.
I see no point in this question, and, especially, in raising a bounty for it.
as they say, "the best is the enemy of good"
You have asked this question ten days ago.
If you'd just go with whatever code you've got, you'd have your tables already and even done with your tests. But you lose so much time just in vain. It's above my understanding.
As for the method you've been asking for (just to keep away all these self-appointed moderators), there are some statements as a food for thought:
mysql's own methods considered more effective in general.
mysql can insert all data from the table into another using INSERT ... SELECT syntax. so, you will need to run only about 30 queries to get your 50 mil records.
and sure mysql can copy whole tables as well.
keep in mind that there should be no indexes at the time of table creation.
I just want to point you to http://www.mysqldumper.net/ which is a tool that allows you to backup and restore big databases with PHP.
The script has some mechanisms to circumvent the maximum execution time of PHP -> imo worth a look.
This is not a solution for generating data, but a great one for importing / exporting.
I have a table
$query=
"CREATE TABLE screenshot ".
"(screenshot_id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, ".
"source_video_id INT UNSIGNED NOT NULL, ".
"screenshot_file_name VARCHAR(128), ".
"x_res INT, ".
"y_res INT, ".
"time INT UNSIGNED);";
mysql_query($query);
Then I insert things into that table.
I often want to do an SQL query and iterate over the result of the query, but end up doing this.
//select all screenshots from video, by video id
$q0=
"SELECT * FROM screenshot ".
"WHERE source_video_id = '$source_video_id' ".
"AND x_res = 120 ".
"AND y_res = 90 ".
"ORDER BY time ASC;";
$r0 = mysql_query($q0);
$n0_num = mysql_numrows($r0);
//for each result
for($n0=0;$n0<$n0_num;$n0++) {
$source_video_id = mysql_result($r0,$n0,'source_video_id');
$time = mysql_result($r0,$n0,'time');
$screenshot_file_name = mysql_result($r0,$n0,'screenshot_file_name');
//do stuff for each returned result!
}
This is just ugly. To get the SQL query results, I have to write this for every column!
$source_video_id = mysql_result($r0,$n0,'source_video_id');
I have to write an ugly loop, get the results for each row returned and do something for each result. Basically I want something like;
foreach($SQL_command) {
//Do for each result
}
I want the column variables for each row to be already set, so that I do not have to do
$source_video_id = mysql_result($r0,$n0,'source_video_id');
For each and every column I want to access!
I am sick of writing boiler plate code to do this for every single table in my data. Are there any frameworks or libraries that would make this less painful?
These are the very basics of a database abstraction layer. It's not hard to program your own, or you can use a generic library like Doctrine or Propel. Every notable PHP framework includes some form of database abstraction as well, you really just need to start using one.
One can suppose I'm a fan of Kohana, but I really love the thing. Get the Kohana 3 and put there the Sprig ORM (it's a fork from original Sprig ORM, but with additional ‘sugar’ :) instead of native Kohana's one. You'll understand how pretty they are together. You'll can access to your tables like this code shows:
//just the basics, updating existing record
$screenshot = Sprig::factory('Screenshot', $id)->load();
$screenshot->x_res = 240;
$screenshot->y_res = 260;
$screenshot->update();
//creating new one
$screenshot = Sprig::factory('Screenshot');
$screenshot->x_res = 300;
$screenshot->y_res = 250;
$screenshot->create();
Additional link to the discussion of the Sprig fork: http://forum.kohanaframework.org/comments.php?DiscussionID=4368
Hope, it'll help you.
If you have the PDO drivers enabled (as you should) you can use the single DB() method as a function from the phunction PHP framework. It was inspired by the DiBi database abstraction layer. The documentation is still underway, but I've posted a short summary in this answer.
function DB($query)
{
static $db = null;
static $result = array();
if (is_null($db) === true)
{
if (preg_match('~^(?:mysql|pgsql):~', $query) > 0)
{
$db = new PDO(preg_replace('~^(mysql|pgsql):(?:/{2})?([-.\w]+)(?::(\d+))?/(\w+)/?$~', '$1:host=$2;port=$3;dbname=$4', $query), func_get_arg(1), func_get_arg(2));
if (preg_match('~^mysql:~', $query) > 0)
{
self::DB('SET time_zone = ?;', 'GMT');
self::DB('SET NAMES ? COLLATE ?;', 'utf8', 'utf8_unicode_ci');
}
}
else if (preg_match('~^(?:sqlite|firebird):~', $query) > 0)
{
$db = new PDO(preg_replace('~^(sqlite|firebird):(?:/{2})?(.+)$~', '$1:$2', $query));
}
}
else if (is_a($db, 'PDO') === true)
{
if (isset($query) === true)
{
$hash = md5($query);
if (empty($result[$hash]) === true)
{
$result[$hash] = $db->prepare($query);
}
if (is_a($result[$hash], 'PDOStatement') === true)
{
if ($result[$hash]->execute(array_slice(func_get_args(), 1)) === true)
{
if (preg_match('~^(?:INSERT|REPLACE)~i', $query) > 0)
{
return $db->lastInsertId();
}
else if (preg_match('~^(?:UPDATE|DELETE)~i', $query) > 0)
{
return $result[$hash]->rowCount();
}
else if (preg_match('~^(?:SELECT|EXPLAIN)~i', $query) > 0)
{
return $result[$hash]->fetchAll(PDO::FETCH_ASSOC);
}
return true;
}
}
return false;
}
}
return $db;
}
Your example query could be written as:
// connect to the MySQL server, do this on your config file or something
DB('mysql://host:port/database_name/', 'username', 'password');
// run the query!
$results = DB('SELECT * FROM screenshot WHERE source_video_id = ? AND x_res = ? AND y_res = ? ORDER BY time ASC;', $source_video_id, 120, 90);
foreach ($results as $result)
{
print_r($result);
}
The above code uses prepared queries which means that you'll also be safe from SQL injection attacks.
PS: I'm biased here, since I'm the developer of the framework. If you run into any problems let me know.
I use RedBean in all my projects and would recommend it without hesitation. The main reasons being:
Minimum configuration required. I don't have to map the database schema into a YAML or JSON file, simply put in the connection parameters and go.
Elegant and easy to understand usage syntax.
Lots of features such as caching and tree relationships.
Pretty good performance.
And here's an example of using it:
$book = R::dispense('book');
$book->title = 'Gifted Programmers';
$book->author = 'Charles Xavier';
$id = R::store($book);