I am trying to track what users are searching for on my site (from a simple search form on the front page) with PHP and MySQL.
At the end of all my queries I am using this query:
INSERT INTO `DiggerActivity_Searches` (
`SearchTerms`,
`SearchType`,
`NumResults`,
`Location`,
`Date`,
`Time`
) VALUES (
'SearchKeywords',
'SearchTypes',
'NumberOfResults',
'User'sLocation',
'CurDate',
'CurTime'
)
Now, whenever there is a new search keyword, it inserts 3 identical rows. However, if I refresh the page it only inserts 1 row, as it should.
The values are passed as a GET like this (I have mod rewritten the URL stuff):
http://www.mysite.com/Search-Category-Search_these_words
You might want to check first whether your script executes the query three times or the script is invoked three times (e.g. by some browser addons).
If you do not have a debugger installed you can use something like
function trace_log() {
static $magic = null;
if (is_null($magic)) {
$magic = uniqid();
}
$s = $magic . ' '. microtime(true) . ":\r\n";
foreach( debug_backtrace() as $d) {
$s .= ' '. $d['file'].'#'.$d['line']."\r\n";
}
file_put_contents('trace_log.txt', $s, FILE_APPEND);
}
...
trace_log();
mysql_query(....) // or stmt->execute() or whatever you use to execute the query.
If the first value of each log entry (the "magic" id) in trace_log.txt changes, your script is invoked multiple times. If it's the same for all three calls to trace_log(), your script executes the query three times.
Your table is missing a primary key. Id suggest a PK like search ID or something similar
Read more about this at Unique key - Wikipedia
Just know you are not alone in dealing with this strange bug.
This same problem showed up on my website in the past few days as well. The only thing I recently added was a third party banner ad.
SOLUTION: I commented out the banner ad script (Hint: from search engine that starts with a G) and everything was good again.
I was just going to make a comment put need more reputation to so...
Anyway, similar happened to me. Turns out I was echoing out debug information above the header causing the browser to reload automatically. Easiest way to check is just use
<script type="text/javascript">alert("loading");</alert>
in the header and see how many times you see it.
Related
Please can somebody be so kind to show me the syntax for using cloneblock in phpword.
So Ive got data in a MySQL DB, and for the single rows that I need to import into my word doc via phpword it works fine....to run my query and search and replace with template processor. BUT, now I want to insert multiple rows into my word document. I've researched and found that the cloneblock method is the answer. However I cannot get it working....currently my code runs but it doesn't seem to get to the second row.
I actually dnt get any error messages. My code executes fine...but the end display word file doesn't display fine....and if you see my code I got an echo statement...which echo's out in my browser exactly what I want "damaged" &"good", (as an example given of one of the row data) but that data doesn't get pulled into my word doc like that...it duplicates "damaged" , "damaged". .
$group_key=1;
do {
//loop to increase my uuid - ($repeatgroup')
$repeatgroup = $id."/"."trailer_repeat_group"."[".$group_key."]";
// query string
$trailer_repeat_grouping = mysqli_query($connect, "SELECT * FROM trailer_repeat_group LEFT JOIN main on trailer_repeat_group.PARENT_KEY = main.metainstanceID WHERE trailer_repeat_group.KEY_id = '$repeatgroup'");
$templateProcessor->cloneBlock('CLONEME', $trailer_count);
while ($row1 = mysqli_fetch_array($trailer_repeat_grouping)) {
//this echo below I am using to test exactly what happends – independent of
//PHPword/templateprocessor
echo $rttc = $row1['right_trailer_tyre_condition'];
//inserting / searching / inserting values
$templateProcessor->setValue("right_trailer_tyre_condition", $rttc);
}
// ending of loop / checking loop
$group_key++;
} while ($group_key <= $trailer_count);
I've done investigation and found the solution.
You're cloning same blocks N times:
$templateProcessor->cloneBlock('CLONEME', $trailer_count);
and then by doing fetch You're trying to replace right_trailer_tyre_condition with some value:
$templateProcessor->setValue("right_trailer_tyre_condition", $rttc);
Issue is that You're replacing all placeholders.
But in fact You need to replace them one by one with different values.
Solution is to define 3rd argument that means count of items to replace.
Simply change it to be:
$templateProcessor->setValue("right_trailer_tyre_condition", $rttc, 1);
After running a few test I realized that my search method does not perform very well if some words of the query is short (2~3 letters).
The way I made the search is by making a MySQL query for every words in the string the visitor entered and then filtering result from each word to see if every words had that result. Once one result has been returned for all words its a match and il show that result to the visitor.
But I was wondering if that's an effective way to do it. Is there any better way while keeping the same functionality ?
Currently the code I have takes about .7Sec making MySQL queries. And the rest of the stuff is under .1Sec.
Normally I would not care much about my search taking .7Sec, But Id like to create a "LiveSearch" and is critical that it loads faster than that.
Here is my code
public static function Search($Query){
$Querys = explode(' ',$Query);
foreach($Querys as $Query)
{
$MatchingRow = \Database\dbCon::$dbCon -> prepare("
SELECT
`Id`
FROM
`product_products` as pp
WHERE
CONCAT(
`Id`,
' ',
(SELECT `Name` FROM `product_brands` WHERE `Id` = pp.BrandId),
' ',
`ModelNumber`,
' ',
`Title`,
IF(`isFreeShipping` = 1 OR `isFreeShippingOnOrder` = 1, ' FreeShipping', '')
)
LIKE :Title;
");
$MatchingRow -> bindValue(':Title','%'.$Query.'%');
try{
$MatchingRow -> execute();
foreach($MatchingRow -> fetchAll(\PDO::FETCH_ASSOC) as $QueryInfo)
$Matchings[$Query][$QueryInfo['Id']] = $QueryInfo['Id'];
}catch(\PDOException $e){
echo 'Error MySQL: '.$e->getMessage();
}
}
$TmpMatch = $Matchings;
$Matches = array_shift(array_values($TmpMatch));
foreach($TmpMatch as $Query)
{
$Matches = array_intersect($Matches,$Query);
}
foreach($Matches as $Match){
$Products[] = new Product($Match);
}
return $Products;
}
As others have already suggested, fulltext search is your friend.
The logic should go more or less like this.
Add a text column to your "product_products" table called, say, "FtSearch"
Write a small script that will run only once, in which you write, for each existing product, the text that has to be searched for into the "FtSearch" column. This is, of course, the text you compose in your query (id + brand name + title and so forth, including the FreeShipping part). You can probably do this with a single SQL statement, but I have no mysql at hand and can't provide you the code for that... you might be able to find it out by yourself.
Create a fulltext index on the "FtSearch" column (doing this after having populated the FtSearch column saves you a little execution time).
Now you have to add the code necessary to ensure that every time any of the fields involved in the search string is inserted/updated, you insert/update the search string as well. Pay attention here, since this includes not only the "Title", "ModelNumber" and "FreeShipping" of the "product_product", but as well the "Name" of the "product_brand". This means that if the name of a product_brand is updated, you will have to regenerate all search strings of all products having that brand. This might be a little slow, depending on how many products are bound to that brand. However I assume it does not happen too often that a brand changes its name, and if it does, it certainly happens in some sort of administration interface where such things are usually acceptable.
At this point you can query the table using the MATCH construct, which is way way faster you could ever get by using your current approach.
Trying to make a little Search feature for a user, so he can type in a date on a webpage made with HTML/PHP, and see which people in the db have registered as member on or after (a date). My user inputs the date in format 2015-10-01. This gets sent to a PHP page with a jqxGrid on it, populated with member details of members conforming to my query on the MySQL database (using PDO).
The query uses the operator >= on a string passed as (for example) "2015-10-01" in the WHERE clause, so I am using STR_TO_DATE to make the comparison work:
WHERE `lastUpdated` >= STR_TO_DATE( ? , '%Y-%m-%d');
With PDO, the ? later gets bound to the date (which was passed in as a string).
The db column for registration date is in DATETIME format, and in the db values look like: "2015-10-12 17:12:52".
My query returns an empty array every time, - and this after many hours of trying every conceivable permutation of date format, both in the MySQL statement and on the page that prepares the data for populating the grid.
Can someone show me what's wrong here?
Thanks!!
SP
Make it
WHERE `lastUpdated` > ?
and check your data and stuff.
Basically, you should never touch PDO until you get raw SQL to work.
okay, so here is the PDO version that works - passing in ? instead of the date:
function getJSONAllMembersByDate($PDOdbObject, $regDate)
{
try
{
$membersByDateSQL = "SELECT `id`, `name_first`, `name_last`, `organization`,`email`, `phone`,`source`,`comments`,`language_id`, `lastUpdated` FROM `member` WHERE lastUpdated>=?";//'$regDate'
$get=$PDOdbObject->prepare($membersByDateSQL);
$get->execute(array($regDate));
$rows = $get->fetchAll(PDO::FETCH_ASSOC);
$json=json_encode($rows);
return $json;
}
The fact that it works proves there were other errors in the file containing the jqxwidget (the version before I posted here). I certainly tried about a million different things to get this working.
I don't know if this counts as an answer, but at least it WORKS! There are so many variables in this problem - json, jqxgrid, pdo ... not forgetting that there are several ways to use PDO. I probably had several errors in different places.
(#apokryfos, the STR_TO_DATE was indeed unnecessary.)
In the end, this is what works:
In the PHP page containing the jqxGrid, the url sent to the server is:
url: 'my-json-responses.php?fct=getJSONAllMembersByDate®Date=<?php echo $fromDate ?>'
This $fromDate comes from the $_POST when the user typed in a date (in the format 2015-10-01) on the input page. When the PHP page containing the jqxGrid loads, it does
$fromDate = $_POST['regDate'];
The url "transits" through the file my-json-reponses.php, which contains many functions. It finds the right one:
if ($_GET['fct'] == 'getJSONAllMembersByDate')
{
$result = getJSONAllMembersByDate($connectionObject, $_GET['regDate']);
echo $result;
}
The $result is called on the file that contains all my PDO database requests, including:
function getJSONAllMembersByDate($PDOdbObject, $regDate) { try
{
$membersByDateSQL = "SELECT `id`, `name_first`, `name_last`, `organization`,`email`, `phone`,`source`,`comments`,`language_id`, `lastUpdated` FROM `member` WHERE lastUpdated>='$regDate'";
$get=$PDOdbObject->query($membersByDateSQL);
$rows = $get->fetchAll(PDO::FETCH_ASSOC);
$json=json_encode($rows);
return $json;
}
catch (PDOException $e)
{
echo "There was a problem getting all members with this search query.";
echo $e->getMessage();
}}
Note that I couldn't make the version using "?" in the query work at all, hence passing in the variable $regDate directly, with single quotes around the variable just to make life interesting.
This returns a nice list of all my users as of 2015-10-01 - but is presumably still open to MySQL injection attacks ...
But after this marathon of debugging I am happy enough for now. (All improvements welcomed, naturally!)
SP
Someone retired in our group and I'm trying to figure out what his merge statement (and associated code) does so I can determine how to convert some (not all) values to integer before sending up. See comments below for questions. I am an absolute newbie with Microsoft SQL and took a class in php a few years ago, but don't have much experience. I've tried googling the merge command but I'm having trouble with a couple parts in it. See my questions below. (// ?)
I've looked at:
http://php.net/manual/en/pdo.query.php
http://stackoverflow.com/questions/4336573/merge-to-target-columns-using-source-rows
http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/index.jsp?topic=%2Fsqlp%2Frbafymerge.htm
I realize these are basic questions but I'm trying to figure it out and nobody around here knows.
function storeData ($form)
{
global $ms_conn, $QEDnamespace;
//I'm not sure what this is doing?? I thought this was where it was sending data up??
$qry = "MERGE INTO visEData AS Target
USING (VALUES (?,?,?,?,?,?,?,?,?,?))
AS Source (TestGUID,pqID, TestUnitID, TestUnitCountID,
ColorID, MeasurementID, ParameterValue,
Comments, EvaluatorID, EvaluationDate)
ON Target.pqID = Source.pqID
AND Target.MeasurementID=Source.MeasurementID //what is this doing?
AND Target.ColorID=Source.ColorID //what is target and source?
WHEN MATCHED THEN
UPDATE SET ParameterValue = Source.ParameterValue,
EvaluatorID = Source.EvaluatorID, //where is evaluatorID and source? My table or table we're send it to?
EvaluationDate = Source.EvaluationDate,
Comments = Source.Comments
WHEN NOT MATCHED BY TARGET THEN
INSERT (TestGUID,
pqID, TestUnitID, TestUnitCountID,
ColorID, MeasurementID,
ParameterValue, Comments,
EvaluatorID, EvaluationDate, TestIndex, TestNumber)
VALUES (Source.TestGUID, Source.pqID,
Source.TestUnitID,
Source.TestUnitCountID,
Source.ColorID, Source.MeasurementID, Source.ParameterValue,
Source.Comments, Source.EvaluatorID, Source.EvaluationDate,?,?);";
$pqID = coverSheetData($form);
$tid = getBaseTest($form['TextField6']);
$testGUID = getTestGUID($tid);
$testIndex = getTestIndex ($testGUID);
foreach ($form['visE']['parameters'] as $parameter=>$element)
{
foreach ($element as $key=>$data)
{
if ( mb_ereg_match('.+evaluation', $key) === true )
{
$testUnitData = getTestUnitData ($form, $key, $tid, $testGUID);
try
{
//I'm not sure if this is where it's sent up??
//Maybe I could add the integer conversion here??
$ms_conn->query ($qry, array(
$testGUID, $pqID,
$testUnitData[0], $testUnitData[1], $testUnitData[2],$element['parameterID'], $data, $element['comments'] $QEDnamespace->userid, date ('Y-m-d'), $testIndex, $tid));
}
catch (Zend_Db_Statement_Sqlsrv_Exception $e)
{
dataLog($e->getMessage());
returnStatus ("Failed at: " . $key);
}
}
}
}
}
This is a bit long for a comment. If you are using SQL Server, then look at the SQL Server documentation on merge. All the SQL Server documentation is on line, and it is very easy to find via Google (and perhaps even easier using Bing).
The purpose of the MERGE command is to do both inserts and updates in one step. Basically, you have a table that has new data ("source") and a table to be updated ("target"). When a record matches, then update the existing record in the target with matching record in source. When a record doesn't match, then insert it into target.
The main advantage of MERGE over two statements is not necessarily the elegant and intuitively obvious syntax. The main advantage is that all the operations occur in a single transaction, so either they all succeed or all fail as one.
The syntax actually isn't that bad. I would recommend that you set up a test database and try a few examples on your own, so you at least understand the syntax. Then, return to this code. When doing so, print out the resulting merge statement and put it in SQL Server Management Studio, where you will have nice color coded key words for the statement. Then go through it step by step, and you'll probably find that it makes lots of sense.
I have an 800mb text file with 18,990,870 lines in it (each line is a record) that I need to pick out certain records, and if there is a match write them into a database.
It is taking an age to work through them, so I wondered if there was a way to do it any quicker?
My PHP is reading a line at a time as follows:
$fp2 = fopen('download/pricing20100714/application_price','r');
if (!$fp2) {echo 'ERROR: Unable to open file.'; exit;}
while (!feof($fp2)) {
$line = stream_get_line($fp2,128,$eoldelimiter); //use 2048 if very long lines
if ($line[0] === '#') continue; //Skip lines that start with #
$field = explode ($delimiter, $line);
list($export_date, $application_id, $retail_price, $currency_code, $storefront_id ) = explode($delimiter, $line);
if ($currency_code == 'USD' and $storefront_id == '143441'){
// does application_id exist?
$application_id = mysql_real_escape_string($application_id);
$query = "SELECT * FROM jos_mt_links WHERE link_id='$application_id';";
$res = mysql_query($query);
if (mysql_num_rows($res) > 0 ) {
echo $application_id . "application id has price of " . $retail_price . "with currency of " . $currency_code. "\n";
} // end if exists in SQL
} else
{
// no, application_id doesn't exist
} // end check for currency and storefront
} // end while statement
fclose($fp2);
At a guess, the performance issue is because it issues a query for each application_id with USD and your storefront.
If space and IO aren't an issue, you might just blindly write all 19M records into a new staging DB table, add indices and then do the matching with a filter?
Don't try to invent the wheel, it's been done. Use a database to search through the file's content. You can looad that file into a staging table in your database and query your data using indexes for fast access if they add value. Most if not all databases have import/loading tools to get a file into the database relatively fast.
19M rows on DB will slow it down if DB was not designed properly. You can still use text files, if it is partitioned properly. Recreating multiple smaller files, based on certain parameters, storing in proper sorted way might work.
Anyway PHP is not the best language for file IO and processing, it is much slower than Java for this task, while plain old C would be one of the fastest for the job. PHP should be restricted to generated dynamic Web output, while core processing should be in Java/C. Ideally it should be Java/C service which generates output, and PHP using that feed to generate HTML output.
You are parsing the input line twice by doing two explodes in a row. I would start by removing the first line:
$field = explode ($delimiter, $line);
list($export_date, ...., $storefront_id ) = explode($delimiter, $line);
Also, if you are only using the query to test for a match based on your condition, don't use SELECT * use something like this:
"SELECT 1 FROM jos_mt_links WHERE link_id='$application_id';"
You could also, as Brandon Horsley suggested, buffer a set of application_id values in an array and modify your select statement to use the IN clause thereby reducing the number of queries you are performing.
Have you tried profiling the code to see where it's spending most of its time? That should always be your first step when trying to diagnose performance problems.
Preprocess with sed and/or awk ?
Databases are built and designed to cope with large amounts of data, PHP isn't. You need to re-evaluate how you are storing the data.
I would dump all the records into a database, then delete the records you don't need. Once you have done that, you can copy those records wherever you want.
As others have mentioned, the expense is likely in your database query. It might be faster to load a batch of records from the file (instead of one at a time) and perform one query to check multiple records.
For example, load 1000 records that match the USD currency and storefront at a time into an array and execute a query like:
'select link_id from jos_mt_links where link_id in (' . implode(',', application_id_array) . ')'
This will return a list of those records that are in the database. Alternatively, you could change the sql to be not in to get a list of those records that are not in the database.