I have a small problem. I am working with some manual testers who are untrained in programming/database design. Our current process means that these manual testers need to insert data into our database at certain times whilst we build a GUI to facilitate this in the future.
In the interim, I would like to create a simple site. What I would like to do with the site is, simply, connect to our database, allow the manual tester to enter some keywords, and return any columns within tables that are close/related to the keywords provided. This would save a lot of time for our testers searching for colums in our (rather large) database.
How could I create a site like this? I think it could be useful for a lot of people, so I have decided to post the question up here to gather the thoughts of StackOverflow.
At the moment, I am thinking a simple PHP page with a textbox, which allows the user to enter some data, separated by commas. Explode the data based on commas, hold it in an array. Connect to my database, then use the Information Schema View to retrieve column information.
My main problem is - what is the most effective way to use the Information Schema View to retrieve columns related to the keywords entered by the users ? How can I ensure the columns returned are the most suitable?
Any input here would be greatly appreciated. Thanks a lot.
Tl;dr is the bolded part, for busy people :)
I think you could achieve this with a simple form and some ajax calls using on key up.
Here is a simple example in which the list will update each time the user enters a letter in the column name they are searching for.
Index.html
<!DOCTYPE html>
<html lang="en">
<head>
<script type="text/javascript">
$(document).ready(function() {
$("#faq_search_input").keyup(function()
{
var faq_search_input = $(this).val();
var dataString = 'keyword='+ faq_search_input;
if(faq_search_input.length>1)
{
$.ajax({
type: "GET",
url: "ajax-search.php",
data: dataString,
success: function(server_response)
{
document.getElementById("searchresultdata").style.display = "block";
$('#searchresultdata').html(server_response).show();
}
});
}return false;
});
});
</script>
</head>
<body>
<div class="searchholder">
<input name="query" class="quicksearch" type="text" id="faq_search_input" />
<div id="searchresultdata" class="searchresults" style="display:none;"> </div>
</div>
</body>
</html>
next we need a script to carry out our search
ajax-search.php
//you must define your database settings
define("DB_HOST", "FOO");
define("DB_USERNAME", "BAR");
define("DB_PASSWORD", "YOUR PASSWORD");
define("DB_NAME", "DATABASE NAME");
if(isset($_GET['keyword']))
{
$search = new mysqli(DB_HOST, DB_USERNAME, DB_PASSWORD, DB_NAME);
if ($search->connect_errno)
{
echo "Failed to connect to MySQL: (" . $search->connect_errno . ") " . $search->connect_error;
$search->close();
}
$keyword = trim($_GET['keyword']) ;
$query ="SELECT COLUMN_NAME FROM ".DB_NAME.".INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_NAME LIKE '%".$keyword."%'";
$values = $search->query($query);
if($values->num_rows != 0)
{
while($row = $values->fetch_assoc())
{
echo $row['COLUMN_NAME']."<br>";
}
}
else
{
echo 'No Results for :"'.$_GET['keyword'].'"';
}
}
As the user types out a column name all of the column name like this will be returned and updated on the fly, without page reload. Hope this helps
You should do something like this:
Form:
<form action="search.php" method="post">
<textarea name="words"></textarea>
<input type="submit">
</form>
search.php
<?php
// You will need a DB user with enough permissions
$link = mysqli_connect($server,$user,$pass);
mysqli_select_db($link,$database_name);
print "<table>";
// Comma separated
$words = explode(",",$_POST['words']);
foreach ($words as $word)
{
$sql = "SELECT COLUMN_NAME FROM ".$database_name.".INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME LIKE '%".$word."%'";
$res = mysqli_query($link,$sql);
while ($row = mysqli_fetch_assoc($res))
{
print "<tr><td>".$row['COLUMN_NAME']."</td></tr>";
}
}
print "</table>";
?>
I can see why you are asking this interesting question. If the tester enters a list of keywords, and you use the information schema view to obtain a list of matching columns, then there is a danger that there will be a lot of false matches that could waste time or cause the tester to enter incorrect information into your system. You want to know how to determine which columns are the best matches to the tester's query. But you want to keep it simple because this is just a temporary workaround, it's not your main application.
The answer is to supplement search results using a reputation-based system. Here is a very simple one that should work well for your application.
First, create two simple tables to store rating information for the tables and columns in your database. Here is the starting structure.
TEST_SEARCH_TABLES:
TABLE_ID
TABLE_NAME
RATING
TEST_SEARCH_COLUMNS:
COLUMN_ID
TABLE_ID
COLUMN_NAME
RATING
Populate TEST_SEARCH_TABLES with the name of every table in your database. Populate TEST_SEARCH_COLUMNS with the name of every column, and link it to the corresponding table. Initialize all the RATING columns to 1000.0 - you will be using the Elo Rating System to supplement your rankings because it is simple, easy to implement and it works great.
When the user enters a list of keywords, don't use Information Schema View. Instead, search the TEST_SEARCH_COLUMNS table for any columns that have any of those keywords. Assign each column a WEIGHT based on the number of hits. (For example, if the search is "customer,amount,income" then a column CUSTOMER_ID would have a weight of 1. A column CUSTOMER_INCOME would have a weight of 2, and CUSTOMER_INCOME_AMOUNT would have a weight of 3.) Calculate the WEIGHT of each table as the sum of the weights of its columns.
Now for each table and column returned by your search, multiply the WEIGHT times the RATING to determine the SEARCH VALUE. Give the tester a list of matching tables in descending order of search value. Within each table, also list the columns in descending order of their search value.
Every time a table or column appears in a search, use the Elo Rating System to give it a WIN against an opponent rated 1000.0. Every time a user selects a column to work with, give both that column and its table a win against an opponent rated 1500.0. In this way, the most useful and successful tables and columns will organically float to the top of your search lists over time.
A side benefit to this approach (using tables instead of information schema view) is that this approach is more extensible. As an enhancement, you could put DESCRIPTION and COMMENTS columns on the TEST_SEARCH_TABLES and TEST_SEARCH_COLUMNS tables, and also search those columns for keyword matches as well.
Here is another optional enhancement - you could put a (+) and (-) button next to each table and column and give it a win against a 2000-rated opponent if the user clicks (+) and a loss against a zero-rated opponent if the user clicks (-). That will allow your testers to vote for columns they find important and to vote against columns that are always getting in the way.
I'm not sure if I fully understood your issue
Take a look at this:
http://php.net/manual/en/function.mysql-list-tables.php
you can get all the tables on a database , store them in an array then filter them using your keywords
I think this could be done in the following steps without any PHP programming and even without need in any web-server.
Write SQL-script which makes everything to retrieve data you need.
Modify script to add columns to result set with simple html-formatting to make you result record like the following:
'<tr><td>', 'resultcolumn1', '</td><td>', 'resultcolumn2','</td></tr>'
Run this script using sqlcmd with output option to file. Give resulting file .html extension.
Place sqlcmd call inside cmd file. After calling sqlcmd call web browser with resulting html file name as parameter. This will display your results to tester.
So, your testers only run cmd file with some parameters and get html page with results. Of course you need to form correct html head and body tags, but this is not a problem.
Now about your main question about how you can be sure the columns returned are the most suitable. I think the most reliable from the most simple ways is to create thesaurus table which contains synonyms for your column names. (This could be done by testers themselves). So you can search your column names from Information Schema View using LIKE in INFORMATION_SCHEMA.COLUMNS as well as in thesaurus table.
Not sure if you want spend time on writing and supporting your solution. For php/mysql I would use http://www.phpmyadmin.net/home_page/index.php or if users can access db directly
http://dev.mysql.com/downloads/gui-tools/5.0.html
Might take some time to tech them how to use it, but will save a lot of problems in a long run.
Another thing, you can create *.sql files that would populate db automatically.
query.sql
CREATE TABLE "example" (
"id" INT NOT NULL AUTO_INCREMENT,
"name" VARCHAR(30),
"age" INT
);
INSERT INTO "example" VALUES
('1', 'a', 1),
('2', 'b', 2);
than you can run it from command line:
mysql -u USER -pPASSWORD database_name < filename.sql
Use mysql_connect method if you use mysql and enter data like:
INSERT INTO tablename
and stuff just read about it.
Related
my database is structured as below
at present the character, groups and vault tables all have data. The games tables is empty however.
Is it possible to write an inner join on the games table that will allow for a php search box to get data from the other three tables even though the games table has no data in it?
Is there another way of doing this that I might not know about?
For example I want to write a php search script for a webpage that searches on the games tables. If a user enters 'Allistair Tenpenny', it will pull the data from the characters table and display it in the search page with the characters name and their history, same with if some one searches a vault it will display the data from the vaults table.
From what I have read of inner joins the data on each joined table must match for it to display. Is there another way to approach this?
No inner join is necessary to get the data you want. You can simply use php to use SQL to search the appropriate table based on the user input.
If there are multiple search fields on your page, just name the submit buttons differently, then have PHP check for the existence of each submit button's POST data from the form, then perform the appropriate search. An example form might be:
<form action="" method="post">
Search Character Name:<input type="text" name="charactername">
<input type="submit" name="charsubmit" value="Search">
</form>
<form action="" method="post">
Search Vault Name:<input type="text" name="vaultname">
<input type="submit" name="vaultsubmit" value="Search">
</form>
Your PHP code can then be structured as:
if (isset($_POST['charsubmit']))
{
$stmt = $db->prepare("SELECT * FROM character_table WHERE character_name = ':mydata'");
$stmt->bindParam(':mydata',$_POST['charactername']);
}
elseif (isset($_POST['vaultsubmit']))
{
$stmt = $db->prepare("SELECT * FROM vault_table WHERE vault_name = ':mydata'");
$stmt->bindParam(':mydata',$_POST['vaultname']);
}
$stmt->execute();
Using prepared statements like this is a good way to prevent SQL injection attacks, thus ensuring user entered data is NEVER put directly into a SQL statement.
Think about it yourself. There should be no redundant information in normalized relative database. All the data tables have a column history. It should not have redundant data, which you could use to identify rows in other tables, but a description of the record set in the own table. We don't know, what vault_number is for. All other tables might be connected to the vault table directly depending on your data structure. Only you know, what your intention building this structure has been.
Further more you have connected games to groups by a redundant column with the group_name. You should avoid any redundancy and give the groups an id. Then connect it to games by a foreign group_id key. Imagine what will happen, when there are 100 games in a group and you decide to rename a group or just want to correct a typo.
Instead of joining the tables why not try to add them to an array?
create an array with the elements you need to display (don't forget to add the same number of elements from all tables (not 3 from 1 table and 4 from the other)
query the first table and use add it to the array
query the the second table and add it to the array
continue adding the tables you need
display the array
Edit:
The array could consist of 4 elements:
vault_number
name
history
type (table it's comming from)
example query would be:
select vault_number, charactor_name as name, history, 'charactor' as type from charactor
select vault_number, group_name as name, history, 'group' as type from groups
You just continue adding this to your array and you can then display all results
Take into consideration what Quasimodo's clone explained about your db normalization and the approach given by Sgt AJ could go a long way.
How can i get all of the records in a table that are out of
sequence so I know which account numbers I can reuse. I have a range
of account numbers from 50100 to 70100. I need to know which account
numbers are not stored in the table (not currently used) so I can use.
For instance say I have the following data in table:
Account Name
------ --------
50100 Test1
50105 Test2
50106 Test4
..
..
..
I should see the results:
50101
50102
50103
50104
because 50101-50104 are available account numbers since not currently in
table.
copied from http://bytes.com/topic/sql-server/answers/78426-get-all-unused-numbers-range
With respect to MYSQL and PHP.
EDITED
My range is 10000000-99999999.
My present way is using MySql query:
'SELECT FLOOR(10000000 + RAND() * 89999999) AS random_number FROM contacts WHERE "random_number" NOT IN (SELECT uid FROM contacts) LIMIT 1';
Thanks.
solution 1:
Generate a table with all possible accountnumbers in it. Then run a query similar to this:
SELECT id FROM allIDs WHERE id NOT IN (SELECT id FROM accounts)
Solution 2:
Get the whole id colummn into an array in php or java orso. Then run a for-loop to check if the number is in the array.
$ids = (array with all ids form the table)
for($i=50100;$i<=70100;$i++){
if(array_search($i, $ids) != -1){
$availableids[] = $i;
}
}
one way would be to create another table - fill it will all allowable numbers, then write a simple query to find the ones in the new table that are not in the original table.
Sort the accounts in the server, and find jumps in PHP while reading in the results. Any jump in the sorted sequence is "free for use", because they are ordered. You can sort with something like SELECT AccountNumber FROM Accounts SORT ASCENDING;.
To improve efficiency, store the free account numbers in another table, and use numbers from this second table until no more remain. This avoids making too many full reads (as in the first paragraph), which may be expensive. While you are at it, you may want to add a hook in the part of the code which deletes accounts, so they are immediately included in this second table, making the first step unnecessary.
I have a mysql table that looks like this:
id author public image1 image2 image3 bio media1 media2 media3 media4 media5 media6
The Field "author" normaly has Firstname (Secondname) Lastname seperated by whitespaces.
How can I sort the array after the Lastname and if just one name is present after this one.
This is the modx query I use to sort after the author but obviously it doesn't use the lastname.
$c = $modx->newQuery('AuthorDe');
$c->sortby('author','ASC');
$authors = $modx->getCollection('AuthorDe',$c);
You're shooting yourself in the foot right now, for a couple of reasons:
When there is only one word in the string, the sorting is hard to predict.
You have indexes for your data for a reason. They make it a lot faster. Using string functions force a table scan. Good enough for 100 data units, slow for 10000 rows and 'database went for a vacation" at 1000000.
Next time you have to use the author field and you realize you have to split it up to words you also have to understand and fix this code snippet on top of the old ones.
That said - I haven't tested it - but try this:
$c->sortby('substring_index(author," ",-1)','ASC');
So to elaborate on the very valuable point jous made, putting multiple points of data in one database column is counter productive.
The sorting you want to do would be simple, fast, & efficient, in a sql query (using the same construct jous showed but without the string operation).
To modify this table you would simply add the following columns to your table in place of author:
firstname
lastname
middlename
To show you how simple this is (and make it even easier) here's the code to do it:
ALTER TABLE [tablename]
ADD COLUMN firstname varchar(32)
ADD COLUMN lastname varchar(32)
ADD COLUMN middlename varchar(32)
DROP COLUMN author;
Then the modx PHP code would be:
$c->sortby('lastname','ASC');
So this is fairly easily done... and if you still need to support other references to author then create a view that returns author in the same way the un-altered table did as shown below (NOTE: you would still have to change the table name reference so it points to the view instead of the table... if this will be a big problem then rename the table and name the view the same as the old table was...):
CREATE VIEW oldtablename AS
SELECT firstname+' '+middlename+' '+lastname' ' AS author
FROM newtablename;
NOTE: if you do create a view like the above then it is probably worth your while to add all of the other columns from the new table (the multiple image & media columns).
NOTE2: I will add, however, that those would ideally be in separate tables with a join table to this one... but if I were in your spot I might agree that expedience might beat utility & future usability.... however if you did put them in different tables you could add those tables to this view (as joins to the new table) and still be able to support existing code that depends on the old table & it's structure.
While the above is all fairly easily done and will work with minor adjustments from you the last part of this is getting your custom table changes to be reflected by xPDO. If you are already comfortable with this and know what to do then great.
If you aren't this is by far the best article on the topic: http://bobsguides.com/custom-db-tables.html
(Yes it is worth getting Bob's code as a snippet so all of this can simply be generated for you once the database changes have been made... (remember you will likely need to delete the existing schema file & xpdo related class files & map files before you run Bob's generation code, or your changes that have the same table name, like the view, won't take effect).
Hope this helps you (or the next person to ask a similar question).
I built a document upload admin screen, where my client can browse and upload pdf documents to a mysql dbase.
I have two separate tables for the Agendas, and one for Minutes. both have separate table names "upload" and "upload_mins".
on the index.php page, I have the page fetch each row of the database, and display all of the valid documents on a "download" page.
I have come a across a problem.
Each dbase is set to increment ID. now that there is two databases, they are coming to utilize the same ID.
so I am pulling from the Agenda table:
http://www.example.com/clerk.php?ID=77
and I am pulling from the Minutes table also:
http://www.example.com/clerk.php?ID=77
and they happen to have the same increment ID.
Is there some way to avoid this? Can I add a field parameter to the minutes to make sure that they don't have the same URL when pulling documents?
Create a integer field, or txt field?
i.e. http://www.example.com/clerk.php?ID=77&min=yes
If these are just documents, you could store them in a single table but have a column called type that differentiates between minutes and agendas. That way, IDs will be unique.
You could also extend the types column be a foreign key to a types table, so you can extend it to include additional document types such as spreadsheets in the future easily.
This would also aid the front-end display, as you would only need one query to fetch the documents within a batch or timeframe, but add them to differing arrays. For example:
// get the last 20 uploaded documents
$sql = "SELECT * FROM documents ORDER BY created DESC LIMIT 0,20";
$res = mysql_query($sql);
while ($doc = mysql_fetch_object($res)) {
$type = $doc->type;
if (!isset($docs[$type]) || !is_array($docs[$type])) {
$docs[$type] = array();
}
$docs[$type][] = $doc;
}
// loop over agendas
foreach ($docs['agendas'] as $agenda) {
echo '' . $agenda->title . '';
}
// loop over minutes
foreach ($docs['minutes'] as $minutes) {
echo '' . $minutes->title . '';
}
...
You say that the problem you are having is with URLs being duplicated. You have 2 different options to solve that.
The first is to create an identifier that tells you which table you want. You could have agenda.php and minutes.php, or you could have clerk.php?ID=77&type=1 and clerk.php?ID=77&type=2.
This will allow you to know which table you are referencing. This is also the approach I would recommend. There is no reason to avoid duplicate keys in different tables as long as you have a way of knowing which table you need.
The second option is to put all your records into a single table. You can then add a type column that specifies what type of document it is. You can then leave your clerk.php file alone, but need to change the upload page to populate the type column.
If you are just storing PDFs in the database, you shouldn't need anything in this table except id, type, and the document itself.
I am working in the confines of a CMS system, which defines certain fields which can be used to make forms for use within the application in PHP.
I am making use of the inputSmartSearch field, which is basically similar to Google suggest.
It allows me to define an SQL query, and then displays records that match as I type in my search.
For my smartsearch, I have chosen it to search through three fields in a different table, and to display those fields concatenated together.
I use define my field like so:
$theinput = new inputSmartSearch($db, "chooseguests", "Choose Guests");
The last parameter is the name of the SQL query to execute.
This works fine, and a guest can be located by searching his last or first name.
However, I have implemented this smartsearch what is meant to be the page to add a sales order.
In each sales order stored in the sales table, distinct from the guest table, I also want to have the firstname and lastname of the guest making the order.
The design of the sales order table has two separate fields for firstname and lastname.
Using smartsearch, I cannot find any way to tokenize the selected input and insert it back into the field.
If I have a smartsearch and can search by firstname or lastname it shows the result as just one fields, and I want it to save the firstname to the firstname field and the lastname to the lastname field.
Each form defined has an include file which defines the function for inserting a record and such, like so:
function prepareVariables($variables){
// if ($variables["webaddress"]=="http://")
// $variables["webaddress"] = NULL;
return $variables;
}
function updateRecord($variables, $modifiedby = NULL, $useUuid = false){
$variables = $this->prepareVariables($variables);
return parent::updateRecord($variables, $modifiedby, $useUuid);
}
function insertRecord($variables, $createdby = NULL, $overrideID = false, $replace = false, $useUuid = false){
$variables = $this->prepareVariables($variables);
return parent::insertRecord($variables, $createdby, $overrideID, $replace, $useUuid);
}
However, I am unsure of how I could modify the insert or update functions to do what I need them to do, or if that is even the correct approach.
Should I be looking for a complex sql query? hidden fields with the content autopopulated from the result of my inputSmartSearch? Something else?
Not sure what's possible with the setup you've got going, but... my first reaction would be to return from the db the names combined with tokens - like - ',' So Smith, John. Then, when getting ready to send back, to split those up ( based upon the known token ). Now where that goes what you've got there, I do not know.
You just need to set the starting value of the smart search manually using the values from the database. Then when the data is posted to the server you can parse and append to the array that is used to store the data in the database. At least that seems like it should work.