Apologies in advance for the wall of text; not sure if this is possible but i thought i'd ask. I've looked online and can't quite find what i want. I have been learning a lot of PHP and MySQL and am at the stage where i am starting to write my own database driven websites. A freely available database i have been practicing with is the eve_sdd_crucible_11 database which is freely available from the game website. I have been using it because it's huge and requires the use of a lot of different skills to get the most out of it.
I would like to do a simple database for exploration. This database queries the main one for information, creates a new table based on the search results and also allows the user to add their own comments on what they have found. I have the various queries ready to go but because i don't want a massive user interface, i want to keep the user side as clean as possible. This app needs to query and display results from the 'mapregions', 'mapconstellations', 'mapsolarsystems' and 'mapdenormalize' tables and insert this information into a new database with the retrieved info plus a comment box for each entity.
Now the preamble is done, this is what i am looking to do:
Query 'mapregions' (region name returns region ID to be used in next query) and display results (linked)->
click on linked result, query 'mapconstellations' (constellation name returns constellation ID to be used in next query) and display results (linked)->
click on linked result, query 'mapsolarsystems' (solar system name returns solar system ID to be used in next query) and display results (linked) ->
click on linked result, query 'mapdenormalise' and display all entities in that system -> Inject content into new database along with comment boxes per listed entity.
Like i said earlier in the post, i have the queries set up ready to go, i have the beginnings of the php for the page but i am stuck on how to link these displayed results to the next query in the chain. All results have to display the name of the entity and it's the entity's corresponding ID number that is used to execute the next query in the chain.
Not sure if i've explained it particularly well, but it's the best i can do at this time of night... Any help or pointers would be vastly appreciated as it's starting to do my head in ;)
Need to look into joins - from reading through that text, it seems like you're missing a basic understanding of how to join tables - your 4th paragraph, to me, sounds like it should be a single query.
Creating a new database and/or table per search shows that you might still be missing some of the fundamentals - as that approach would never scale and would be a nightmare to manage.
Start reading up on mysql joins: Mysql Joins and go from there, looking at other examples of how joins work and real world examples - that will hugely affect how you continue building this.
Related
I'm going over my old system and updating the old, deprecated code from mysql to PDO and just giving it a general makeover. The system basically grabs data from the database, displays it in a fancy table and its then sortable by clicking on the table headers or by changing the search filters by the side. (e.g having a drop down box of different categories)
It's been a right nightmare updating the old system because the code is just all over the place. It currently gets the search filters information from the URL via GET and dynamically creates a query string. This makes it even more complicated when working with prepared statements.
Is there anything out there that can help me avoid a headache? I'm looking for something that allows me to display information from a database and also has the options to add search filters. I could probably build one from scratch but I just don't want to waste the effort on something that has (probably) already been solved.
I personally like using datatables. It can process large datasets very quickly and all of that sorting you talked about is a feature with datatables.
https://datatables.net/
I'm programming a search engine for my website in PHP, SQL and JQuery. I have experience in adding autocomplete with existing data in the database (i.e. searching article titles). But what about if I want to use the most common search queries that the users type, something similar to the one Google has, without having so much users to contribute to the creation of the data (most common queries)? Is there some kind of open-source SQL table with autocomplete data in it or something similar?
As of now use the static data that you have for auto complete.
Create another table in your database to store the actual user queries. The schema of the table can be <queryID, query, count> where count is incremented each time same query is supplied by some other user [Kind of Rank]. N-Gram Index (so that you could also auto-complete something like "Manchester United" when person just types "United", i.e. not just with the starting string) the queries and simply return the top N after sorting using count.
The above table will gradually keep on improving as and when your user base starts increasing.
One more thing, the Algorithm for accomplishing your task is pretty simple. However the real challenge lies in returning the data to be displayed in fraction of seconds. So when your query database/store size increases then you can use a search engine like Solr/Sphinx to search for you which will be pretty fast in returning back the results to be rendered.
You can use Lucene Search Engiine for this functionality.Refer this link
or you may also give look to Lucene Solr Autocomplete...
Google has (and having) thousands of entries which are arranged according to (day, time, geolocation, language....) and it is increasing by the entries of users, whenever user types a word the system checks the table of "mostly used words belonged to that location+day+time" + (if no answer) then "general words". So for that you should categorize every word entered by users, or make general word-relation table of you database, where the most suitable searched answer will be referenced to.
Yesterday I stumbled on something that answered my question. Google draws autocomplete suggestions from this XML file, so it is wise to use it if you have little users to create your own database with keywords:
http://google.com/complete/search?q=[keyword]&output=toolbar
Just replacing [keyword] with some word will give suggestions about that word then the taks is just to parse the returned xml and format the output to suit your needs.
I'm building a site that contains "panels" which are used as containers for various information. I have set it up so panels are editable, which is simple for panels that just contain text. For that I just grab the content from the database and wrap it in a textarea rather than a <p> tag. For panels that contain table views however this is proving to be a more difficult task.
First off I'm having trouble allowing the admin of the site pick what information is in a given table (for example if the admin wanted to add a panel view that showed each members first name, last name, and picture they could pick from those columns in my database). I've come up with a few ways to do this, but each have their own set of problems.
I tried using the INFORMATION_SCHEMA table to generate a table containing the possible tables and columns that the user can choose from. But when it comes to building the query with PDO I have problems. For instance with prepared statements you can't use a variable for the schema.
I also thought of using MySQL views but I can't seem to figure out how to do it that way either.
My second problem is allowing the admin to add rows to the tables directly. Right now all the add row template does is create a row with a text field in each column. This is good for purely text options (like first name) but for things like pictures obviously a text field won't work. Should I create a table that contains this metadata or perform the check in PHP? If it's the latter, how would I know what input type the column needs?
I think my main problem is I'm trying to solve too many things with only one design change (or not focusing on one problem at a time). It's resulting in me becoming very flustered and confused. Help is greatly appreciated and if you need anymore information like how my database tables are currently setup I'll provide an ERD.
Edit: I just wanted to make it clear that I don't want to allow the user to actually manipulate the tables in the database, but rather select what information from the existing tables is shown on a given panel.
Coding the ability for users to freely query a database has a lot of problems (including security) and is way more complicated than predefined information queries that simply return a defined set of information.
It also places the burden of defining which info might be useful onto the user. It places the burden of deciding whether a certain information should be accessible to a particular user onto the query logic and database access rules.
Effectively you are trying to copy PHPMyAdmin with a different design and only your defined database as a target.
What are some of the techniques for providing personalized search results to a logged in user? One way I can think of will be by analyzing the user's browsing history.
Tracking: A log of a user's activities like pages viewed and 'like' buttons clicked can be use to bias search results.
Question 1: How do you track a user's browsing history? A table with columns user_id, number_of_hits, page id? If I have 1000 daily visitors, each browsing 10 pages on average, wont there be a large number of records to select each time a personalized recommendation is required? The table will grow at 300K rows a month! It will take longer and longer to select the rows each time a search is made. I guess the table for recording 'likes' will take the same table design.
Question 2: How do you bias the results of a search? For example, if a user as been searching for apple products, how does the search engine realise that the user likes apple products and subsequently bias the search towards them? Tag the pages and accumulate a record of tags on the page visited?
You probably don't want to use a relational database for this type of thing, take a look at mongodb or cassandra. That's because you basically want to add a new column to the user's history so a column-oriented database makes more sense.
300k rows per month is not really that much, in fact, that's almost nothing. it doesn't matter if you use a relational or non-relational database for this.
Straightforward approach is the following:
put entries into the table/collection like this:
timestamp, user, action, misc information
(make sure that you put as much information as possible, such that you don't need to join this data warehousing table with any other table)
partition by timestamp (one partition per month)
never go against this table directly, instead have say daily report jobs running over all data and collect and compute the necessary statistics and write them to a summary table.
reflect on your report queries and put appropriate partition local indexes
only go against the summary table from your web frontend
If you stored only the last X results as opposed to everything, it would probably be do-able. Might slow things down, but it'd work. Any time you're writing more data and reading more data, there's going to be an impact. Proper DBA methods such as indexing and query optimizing can help, but no matter what you use there's going to be an affect.
I'd personally look at storing just a default view for the user in a DB and use the session to keep track of the rest. Sure, when you login there'd be no history. But you could take advantage of that to highlight a set of special pages that you think are important or relevant to steer the user to. A highlight system of sorts. Faster, easier, and more user-friendly.
As for bias, you could write a set of keywords for each record and array sort them accordingly. Wouldn't be terribly difficult using PHP.
I use MySQL and over 2M records (page views) a month and we run reports on that table daily and often.
The table is partitioned by month (like already suggested) and indexed where needed.
I also clear the table from data that is over 6 months by creating a new table called "page_view_YYMM" (YY=year, MM=month) and using some UNIONS when necessary
for the second question, the way I would approach it is by creating a table with the list of your products that is a simple:
url, description
the description will be a tag stripped of the content of your page or item (depend how you want to influence the search) and then add a full text index on description and a search on that table adding possible extra terms that you have been collecting while the user was surfing your site that you think are relevant (for example category name, or brand)
Google unfortunately didn't seem to have the answers I wanted. I currently own a small search engine website for specific content using PHP GET.
I want to add a latest searches page, meaning to have each search recorded, saved, and then displayed on another page, with the "most searched" at the top, or even the "latest search" at the top.
In short: Store my latest searches in a MySQL database (or anything that'll work), and display them on a page afterwards.
I'm guessing this would best be accomplished with MySQL, and then I'd like to output it in to PHP.
Any help is greatly appreciated.
Recent searches could be abused easily. All I have to do is to go onto your site and search for "your site sucks" or worse and they've essentially defaced your site. I'd really think about adding that feature.
In terms of building the most popular searches and scaling it nicely I'd recommend:
Log queries somewhere. Could be a MySQL db table but a logfile would be more sensible as it's a log.
Run a script/job periodically to extract/group data from the log
Have that periodic script job populate some table with the most popular searches
I like this approach because:
A backend script does all of the hard work - there's no GROUP BY, etc made by user requests
You can introduce filtering or any other logic to the backend script and it doesn't effect user requests
You don't ever need to put big volumes of data into the database
Create a database, create a table (for example recent_searches) and fields such as query (the query searched) and timestamp (unix timestamp that the query was made) said, then for your script your MySQL query will be something like:
SELECT * FROM `recent_searches` ORDER BY `timestamp` DESC LIMIT 0, 5
This should return the 5 most recent searches, with the most recent one appearing first.
Create table (something named like latest_searches) with fields query, searched_count, results_count.
Then after each search (if results_count>0), check, if this search query exists in that table. And update or insert new line into table.
And on some page you can just use data from this table.
It's pretty simple.
Ok, your question is not yet clear. But I'm guessing that you mean you want to READ the latest results first.
To achieve this, follow these steps:
When storing the results use an extra field to hold DATETIME. So your insert query will look like this:
Insert into Table (SearchItem, When) Values ($strSearchItem, Now() )
When retrieving, make sure you include an order by like this:
Select * from Table Order by When Desc
I hope this is what you meant to do :)
You simply store the link and name of the link/search in MySQL and then add a timestamp to record what time sb searched for them. Then you pull them out of the DB ordered by the timestamp and display them on the website with PHP.
Create a table with three rows: search link timestamp.
Then write a PHP script to insert rows when needed (this is done when the user actually searches)
Your main page where you want stuff to be displayed simply gets the data back out and puts them into a link container $nameOfWebsite
It's probably best to use a for/while loop to do step 3
You could additionally add sth like a counter to know what searches are the most popular / this would be another field in MySQL and you just keep updating it (increasing it by one, but limited to the IP)