I'm building a custom IPN handler in PHP, which validates the data Paypal sends in a postback, and then inserts some of it into an sqlite database.
Every part of the script has worked before.
Transactions get reported as valid, and logged. Everything works, up to line 32, which calls a custom function.
Here's line 32:
$sql = insert_query_builder("PP_Data", $keywords);
and here's how the function's defined, in a require()'d file:
function insert_query_builder($table, $keywords, $info = NULL) {
$sql_1 = "INSERT INTO $table (";
$sql_2 = ") VALUES (";
foreach ( $keywords as $num => $keyword ) {
if (isset($info)) {
if ($num != (count($keywords - 1))) {
$sql_2 .= $db->quote($info[$keyword]) . ", ";
$sql_1 .= $keyword . ", ";
} else {
$sql_2 .= $db->quote($info[$keyword]) . ")";
$sql_1 .= $keyword;
}
} else {
if ($num != (count($keywords - 1))) {
$sql_2 .= $db->quote($_POST[$keyword]) . ", ";
$sql_1 .= $keyword . ", ";
} else {
$sql_2 .= $db->quote($_POST[$keyword]) . ")";
$sql_1 .= $keyword;
}
}
}
$sql = $sql_1 . $_sql_2;
return $sql;
}
This code has worked. the SQL statement it generates is then used on the DB, and it's responsible for one of the rows in the DB.
Even so, now it fails silently. The script works, otherwise.
here you can find complete copies of all three files used, and the log file.
As you can see, it's not writing anything to the log after the custom function
and, when i put that last working file_put_contents() just after it, it stops working.
Any ideas? doing a php -f IPNrx.php gives no output, and logs an invalid transaction, like it should, but it doesn't test that branch of the logic.
EDIT: there are three problems, two of them fatal, one insiduous.
First, the if statements should read if ($num != (count($keywords) - 1)), instead of if ($num != (count($keywords - 1)))
Second, the $db object is not inside the function's scope, so i need to add global $db at the top of the function
Third, there's an extra underscore near the end; $sql = $sql_1 . $_sql_2; should be $sql = $sql_1 . $sql_2;
Thanks a million guys. i'd never have found these without your help.
This line references a non-existent variable:
$sql = $sql_1 . $_sql_2;
Should be:
$sql = $sql_1 . $sql_2;
Therefore, the query is invalid. You should check the output after executing the SQL query (mysql_error(), for example).
Related
I'm trying to run a SQL query, and it is working correctly in phpMyAdmin, but whe running it in PHP, the query comes back very wonky. The following query yields two different results:
SELECT `stock_ticker`, `stock_simpleName`, `stock_logo`
FROM `stocks`
WHERE stock_simpleName REGEXP'^c'
I get the following results in phpMyAdmin (Which is correct):
stock_simpleName
----------------------
Coca-Cola
Campbell's
ConAgra Foods
However, in PHP it comes out really weird:
stock_simpleName
-----------------------
Coca-Cola
MasterCard
Campbell's
Microsoft
The Walt Disney Company
PepsiCo
The Hershey Company
Proctor & Gamble
ConAgra Foods
...etc...
Why is this happening? This doesn't make any sense. Is it due to a server setting in PHP or some form of encoding or whatnot?
EDIT:
Here is my PHP Code:
The sub-model class (the creator of the pieces):
public function allOtherSearchResults($query, $dontQuery = null) {
$name = "stocks";
$where = "stock_simpleName REGEXP'^" . $query . "'";
$cols = array("stock_ticker", "stock_simpleName", "stock_logo");
$limit = 5;
return $this->select($name, $cols, $where, $limit);
}
The main-model class (this runs the query):
public function select($tableName, $columns, $where = null, $limit = null) {
global $purifier;
// Make columns SQL friendly
$cols = "`";
$cols .= implode("`, `", $columns);
$cols .= "`";
$table = "`" . $tableName . "`";
if (!empty($where)) {
$where = " WHERE " . $where;
}
// Check limit
if (!empty($limit)) {
$limit = " LIMIT $limit";
}
// SQL CODE
$sql = "SELECT " . $cols . " FROM " . $table . $where . $limit;
// SQL DEBUGGING IF CODE RETURNS BOOLEAN ERROR
echo $sql . "<br>";
$query = $this->conn->query($sql);
// Store the value in a variable called table with an array of that table's name followed by it's values
// EX: $model->table["bands"]["band_name"]
//
// Accessible by the individual page/directory's controller's
while($row = $query->fetch_assoc()){
// Store values as $model->table["tableName"]["columnName"]["index (usually 0)"]
foreach ($row as $key => $val) {
$this->data[$tableName][$key][] = $row[$key];
}
}
// Loop through results to clean them
// Foreach loops through each column
// Make sure the table isn't empty (i.e. login returns an error)
if (!empty($this->data[$tableName])) {
foreach ($this->data[$tableName] as $key => $tableArray) {
// For loop goes through each value in a certain row
for ($i = 0; $i < count($tableArray); $i++) {
// Convert from data variable to table after HTML PURIFIER
$this->table[$tableName][$key][$i] = $purifier->purify($tableArray[$i]);
}
}
}
// Declare the array after loop has finished for use in view
$this->table;
if (!empty($this->table)) {
return true;
}
}
And it gives me the same SQL output as above. I am not sure if there is a different interpretation of certain characters in PHP versus the standard MySQL in phpMyAdmin. Has anyone even had this problem before?
I'm guessing, that there is a problem wiht ^ character.
Try to set proper connection & result encoding, eq.
$this->conn->query("MYSQL SET NAMES utf8");
$this->conn->query("MYSQL SET CHARACTER SET utf8");
Also, check if your php script file is saved in UTF-8 encoding.
Moreover, you should consider of using prepared statement (even to prevent SQL Injection):
$this->conn->prepare("SELECT * FROM `stocks` WHERE `stock_simpleName` REGEXP ?");
$this->conn->bind_param("s", "^c");
$this->conn->execute();
$query = $this->conn->get_result();
<?php
if(isset($_POST['submit'])) {
$fields = array('field1', 'field2', 'field3');
$conditions = array();
foreach($fields as $field){
if(isset($_POST[$field]) && $_POST[$field] != '') {
$conditions[] = "`".$field."` like '%" . mysql_real_escape_string($_POST[$field]) . "%'";
}
}
$query = "SELECT * FROM customer ";
if(count($conditions) > 0) {
$query .= "WHERE " . implode (' AND ', $conditions);
}
$result = mysql_query($query);
$say = mysql_num_rows($result);
if ($say == 0) {
echo "<tr>no result.</tr>";
} else {
echo '...';
while($row = mysql_fetch_array($result))
{
...
}}
} ?>
Why doesn't this code checking empty fields? It returns results that has empty field even form submits empty.
The only improvement I think of is trim():
if(isset($_POST[$field]) && trim($_POST[$field]) != '') {
however, I am sure it is not the issue.
Have you ever thought of printing the resulting query out?
Look, you're writing a program to create some string (SQL query). But for some reason never interested in this program's direct result, judging it by some indirect results. May be it's data/query logic makes such results, but the query itself is okay?
if the query is still wrong - continue debugging.
Echo everything involved - print variables, condition results, intermediate results in the loop - and look for inconsistencies
$query = "SELECT * FROM customer ";
if(count($conditions) > 0) {
$query .= "WHERE " . implode (' AND ', $conditions);
}
When form is submitted empty ($conditions=0) it returns all table (select * from customer).
Added an else condition and fixed. Thanks for print query advices.
For checking something is empty or not. You can use empty() method.
Check this:
empty()
isset() only check whether that object/variable is set or not. For more details check this
isset()
Generating the statement elements
I am trying to create an MSSQL prepared statement in a generic way.
Basically, I loop trough the fields, and add them to the prepared SQL string and to the reference parameter (scroll down to see the code).
This results in (names removed):
Prepared query: :
INSERT INTO table
([field],[field],[field],[field],[field],[field],[field],[field],[field],[field],[field],[field],[field],[field],[field],VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
and
sPrepStatement:
return array(&$aLine[0],&$aLine[1],&$aLine[2],&$aLine[3],&$aLine[4],&$aLine[5],&$aLine[6],&$aLine[7],&$aLine[8],&$aLine[9],&$aLine[10],&$aLine[11],&$aLine[12],&$aLine[13],&$aLine[14]);
Preparing the statement
I've tried the following 4 approaches to get this to work with the sqlsrv_prepare statement:
$oSQLStmnt = sqlsrv_prepare($dbhandle, $sPreppedSQL, eval($sPrepStatement));
$fReturnPrepVals = function(){ return eval($sPrepStatement); } ;
$oSQLStmnt = sqlsrv_prepare($dbhandle, $sPreppedSQL, $fReturnPrepVals);
$oSQLStmnt = sqlsrv_prepare($dbhandle, $sPreppedSQL,$aPrepValues);
$oSQLStmnt = call_user_func_array('sqlsrv_prepare',array($dbhandle,$sPreppedSQL, eval($sPrepStatement)));
This either does not work, or inserts blanks into the database.
This is the loop that executes the SQL:
foreach ($aLines as $iLineNum => $sLine) {
$aLine = explode('|', $sLine);
print_r($aPrepValues);
print_r(eval($sPrepStatement));
sqlsrv_execute($oSQLStmnt);
}
The eval($sPrepStatement) works just fine, which makes sense. But I assume it is 'parsed too early' to work in the prepare statement. It should actually only be parsed as the query is executed, but I'm at a loss how to achieve that.
Code for the generation of sql statements:
<?php
//Setup the basic query
$sSql = "INSERT INTO " . $sQName . "\n";
$sFieldNames = "(";
$sValues = "VALUES(";
//This will be evalled to prepare the query, so inserting will be easy.
$sPrepStatement = "return array(";
//FieldIndex to keep track of the current column number
$iFI = 0;
$aLine = array();
foreach ($aSQLQueries[$sQName]['fields'] as $sFieldName => $sQ) {
$sFieldNames .= "[" . $sFieldName . "],";
$sValues .= "?,";
//Init the $aLine var to prevent errors
$aLine[$iFI] = '';
//This will be evalled, so no "" as that would parse it directly
//The values wil be passed by reference
$sPrepStatement .= '&$aLine[' . $iFI . '],';
//Different approach
$aPrepValues[] = &$aLine[$iFI];
$iFI++;
}
$sFieldNames = substr($sFieldNames, 0, -1) . ")";
$sValues = substr($sValues, 0, -1) . ")";
$sPrepStatement = substr($sPrepStatement, 0, -1) . ");";
$sPreppedSQL = $sSql . $sFieldNames . " " . $sValues;
I ended up making it work with PDO, only to find that it could not handle millions of imports at a time, so I eventually used CSV import
I didn't expect this script (throw-away) to be leaking and I haven't figured out what the culprit is. Can you spot anything? Although this is throw-away code, I'm concerned that I'll repeat this in the future. I've never had to manage memory in PHP, but with the number of rows in the db, it's blowing up my php instance (already upped the memory to 1Gb).
The california table is especially larger than the others (currently 2.2m rows, less as I delete duplicate rows). I get a memory error on line 31 ($row = mysql_fetch_assoc($res))
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried
to allocat e 24 bytes) in C:\Documents and Settings\R\My Documents\My
Webpages\cdiac\cdiac_ dup.php on line 31
PHP 5.3.0, mysql 5.1.36. part of a wamp install.
here's the entire code. the purpose of this script is to delete duplicate entries (data was acquired into segmented tables, which was far faster at the time, but now I have to merge those tables.)
what's causing it? something I'm overlooking? or do I just need to watch the memory size and call garbage collection manually when it gets big?
<?php
define('DBSERVER', 'localhost');
define('DBNAME', '---');
define('DBUSERNAME', '---');
define('DBPASSWORD', '---');
$dblink = mysql_connect(DBSERVER, DBUSERNAME, DBPASSWORD);
mysql_select_db(DBNAME, $dblink);
$state = "AL";
//if (isset($_GET['state'])) $state=mysql_real_escape_string($_GET['state']);
if (isset($argv[1]) ) $state = $argv[1];
echo "Scanning $state\n\n";
// interate through listing of a state to check for duplicate entries (same station_id, year, month, day)
$DBTABLE = "cdiac_data_". $state;
$query = "select * from $DBTABLE ";
$query .= " order by station_id, year, month, day ";
$res = mysql_query($query) or die ("could not run query '$query': " . mysql_errno() . " " . mysql_error());
$last = "";
$prev_row;
$i = 1;
$counter = 0;
echo ".\n";
while ($row = mysql_fetch_assoc($res)) {
$current = $row["station_id"] . "_" . $row["year"] . "_" . sprintf("%02d",$row["month"]) . "_" . sprintf("%02d",$row["day"]);
echo str_repeat(chr(8), 80) . "$i $current ";
if ($last == $current) {
//echo implode(', ', $row) . "\n";
// merge $row and $prev_row
// data_id station_id, state_abbrev, year, month, day, TMIN, TMIN_flags, TMAX, TMAX_flags, PRCP, PRCP_flags, SNOW, SNOW_flags, SNWD, SNWD_flags
printf("%-13s %8s %8s\n", "data_id:", $prev_row["data_id"], $row["data_id"]);
if ($prev_row["data_id"] == $row["data_id"]) echo " + ";
$set = "";
if (!$prev_row["TMIN"] && $row["TMIN"]) $set .= "TMIN = " . $row["TMIN"] . ", ";
if (!$prev_row["TMIN_flags"] && $row["TMIN_flags"]) $set .= "TMIN_flags = '" . $row["TMIN_flags"] . "', ";
if (!$prev_row["TMAX"] && $row["TMAX"]) $set .= "TMAX = " . $row["TMAX"] . ", ";
if (!$prev_row["TMAX_flags"] && $row["TMAX_flags"]) $set .= "TMAX_flags = '" . $row["TMAX_flags"] . "', ";
if (!$prev_row["PRCP"] && $row["PRCP"]) $set .= "PRCP = " . $row["PRCP"] . ", ";
if (!$prev_row["PRCP_flags"] && $row["PRCP_flags"]) $set .= "PRCP_flags = '" . $row["PRCP_flags"] . "', ";
if (!$prev_row["SNOW"] && $row["SNOW"]) $set .= "SNOW = " . $row["SNOW"] . ", ";
if (!$prev_row["SNOW_flags"] && $row["SNOW_flags"]) $set .= "SNOW_flags = '" . $row["SNOW_flags"] . "', ";
if (!$prev_row["SNWD"] && $row["SNWD"]) $set .= "SNWD = " . $row["SNWD"] . ", ";
if (!$prev_row["SNWD_flags"] && $row["SNWD_flags"]) $set .= "SNWD_flags = '" . $row["SNWD_flags"] . "', ";
$delete = "";
$update = "";
if ($set = substr_replace( $set, "", -2 )) $update = "UPDATE $DBTABLE SET $set WHERE data_id=".$prev_row["data_id"]." and year=".$row["year"]." and month=".$row["month"]." and day=".$row["day"].";\n";
if ($row["data_id"] != $prev_row["data_id"]) $delete = "delete from $DBTABLE where data_id=".$row["data_id"]." and year=".$row["year"]." and month=".$row["month"]." and day=".$row["day"].";\n\n";
if ($update) {
$r = mysql_query($update) or die ("could not run query '$update' \n".mysql_error());
}
if ($delete) {
$r = mysql_query($delete) or die ("could not run query '$delete' \n".mysql_error());
}
//if ($counter++ > 5) exit(0);
}
else {
$last = $current;
unset($prev_row);
//copy $row to $prev_row
foreach ($row as $key => $val) $prev_row[$key] = $val;
}
$i++;
}
echo "\n\nDONE\n";
?>
I would try two things:
1) Instead of running the UPDATE and DELETE queries inside the loop using mysql_query, save them in a text file, to execute later. For example: file_put_contents('queries.sql', $update, FILE_APPEND );
2) Instead of doing everything inside the while ($row = mysql_fetch_assoc($res)) loop, first grab all SELECT query results, then close database connection freeing all database resources, including the query result. Only after this, perform the loop process.
If you run out of memory while storing the database results in one array, you can try saving the results in a temporary file instead (one record per line / FILE_APPEND), and then use this file in the loop (reading one line per record, using fgets function).
Work smarter, not harder:
SELECT station_id, year, month FROM table
GROUP BY station_id, year, month
HAVING COUNT(*) > 1
That'll get you all the station_id/year/month tuples that appear in the table more than once. Assuming that most of your data is not duplicates, that'll save you a lot of memory, since now you just have to go through these tuples and fix up the rows matching them.
I found this when trying to trace down a memory use problem on a script of mine. Having solved the issue for mine I thought it worth adding a reply here for the next person who comes along with the same issue.
I was using mysqli, but much the same applies for mysql.
The problem I found was the queries not freeing their results. The solution was to use mysqli_free_result() after executing the update and delete queries. But more importantly on the mysqli_query for the loop I used the extra parameter of *MYSQLI_USE_RESULT* . There are side effects of this, so use a separate connection for the update and delete queries.
I have to build a query based on certain conditions. Is there a better way of doing it than the way I have done below? It works fine but I can see it getting out of hand fairly quickly if there were more conditions since I check if any previous conditions had been met every time I check a new one.
$sql = "SELECT DISTINCT fkRespondentID FROM tblRespondentDayTime";
if (!empty($day) || !empty($time) || !empty($sportID)) {
$sql .= " WHERE";
if (!empty($day)) {
$sql .= " fldDay='$day'";
}
if (!empty($time)) {
if (!empty($day)) {
$sql .= " AND";
}
$sql .= " fldTime='$time'";
}
if (!empty($sportID)) {
if (!empty($day) || !empty($time)) {
$sql .= " AND";
}
$sql .= " fkRespondentID IN (SELECT fkRespondentID FROM tblRespondentSport WHERE fkSportID='$sportID')";
}
}
I would use the old "WHERE 1=1" trick; add this as the first condition, and then you can assume the "AND" condition on each statement that follows.
$sql = "SELECT DISTINCT fkRespondentID FROM tblRespondentDayTime WHERE 1=1";
if (!empty($day))
$sql .= "AND fldDay='$day'";
if (!empty($time)) {
$sql .= "AND fldTime='$time'";
if (!empty($sportID))
$sql .= "AND fkRespondentID IN (SELECT fkRespondentID FROM tblRespondentSport WHERE fkSportID='$sportID')";
Build a list/array of conditions, where each conditional is optional (i.e. if condition is valid, push it on the list).
If this list is > 0, add "where" and then add the list join'ed by "and".
Rather than doing checks like if (!empty($day) || !empty($time)) you can create a $whereClause variable and check it like this:
$sql = "SELECT DISTINCT fkRespondentID
FROM tblRespondentDayTime";
$whereClause = '';
// fldDay
if (!empty($day)) {
$whereClause .= " fldDay='$day'";
}
// fldTime
if (!empty($time)) {
if (!empty($whereClause)) {
$whereClause .= ' AND ';
}
$whereClause .= " fldTime='$time'";
}
// fkRespondentID
if (!empty($sportID)) {
if (!empty($whereClause)) {
$whereClause .= ' AND ';
}
$whereClause .= " fkRespondentID IN (SELECT fkRespondentID
FROM tblRespondentSport
WHERE fkSportID='$sportID')";
}
if (!empty($whereClause)) {
$whereClause = ' WHERE '.$whereClause;
}
$sql .= $whereClause;
This will also work if you need to, say, change some to an OR (1=1 trick won't work in that case and could even prove quite hazardous).
you could try putting your variables in an array and having a boolean that tells if you need to add the "AND" in before your next phrase. This would shorten your control statements to a foreach and a nested if.
Here is my solution:
$sql = "SELECT * FROM table";
$conditions = array(
'fldDay' => $day,
'fldTime' => $time,
);
if (count(array_filter($conditions))) {
$sql .= ' WHERE ';
$sql .= implode(' AND ', array_map(function($field, $value) {
return $field . '=\'' . pg_escape_string($value) . '\'';
}, array_keys($conditions), $conditions));
}
Please note, that because of the closures, this won't work below PHP 5.3. If you are using an older PHP, make the closure as a separate function, or substitute it with a foreach.
Unfortunately, building dynamic SQL is a tedious experience and even if you can change a few things in your logic (which actually looks relatively clean), it's still going to be ugly.
Fortunately, Object-relational mapping exists. I'm not too familiar with PHP, but Perl has several CPAN modules such as SQL::Abstract, which will allow you to build fairly complex SQL statements using basic data structures.
If you use stored procedures, you can do something like this:
CREATE PROCEDURE `FindRespondents` (
IN `_day` varchar(255),
...
)
BEGIN
SELECT DISTINCT fkRespondentID
FROM tblRespondentDayTime
WHERE (_day Is Null OR fldDay = _day)
AND ...
END;
|
Passing in null for _day means any fldDay is OK. Any other value for _day, and it must be matched. I assumed fldDay is text, but of course you can type everything properly here.
I know some people aren't fans of stored procedures, but it can be handy encapsulate query logic this way.