I'm having trouble figuring out an efficient way to register a big number of data in my database, using php/mysql.
I have a list of phone numbers, which are stored in a number table.
PHONE_ID | PHONE_NUMBER | COUNTRY_ID
varchar
And a table linking a phone number with a "contact list group". Let's assume it's name is link_phonegroup.
For each phone/group link, I have a parameters list.
GROUP_ID | PHONE_ID | PARAMETERS_LIST
varchar
My script should be able to, given a group_id, compute through millions of numbers and :
retrieve the number_id associated with the number OR insert the number if it does not exist (and return the insert_id)
update the parameters_list associated with this group/number pair and if this pair does not exist, insert it
Currently, I loop (foreach) through my numbers, and, foreach number :
SELECT PHONE_ID FROM number WHERE PHONE_NUMBER = '$number'
If it does not exist, I INSERT INTO ... and retrieve this newly created id with mysql_insert_id()
Then, with this ID, I SELECT PARAMETERS_LIST FROM link_phonegroup WHERE GROUPE_ID = $group_id AND PHONE_ID = $phone_id.
If it does exist, I then UPDATE my parameters list, and if it does not, I INSERT a new row inside my link_phonegroup table.
My problem, as you may imagine, is that for each of my X millions numbers, I will fire 4 queries. That is slow, inefficient, and scary.
I learned about the INSERT INTO ON DUPLICATE KEY technique (MySQL manual page). My tests were super-slow, and i gave up.
I learned about the UPDATE CASE WHEN technique (Example).
Basically, my current goal is to fire ONE query each 200-ish loop (200 is a random number here, i'll do some tests with other values), which would insert/update/retrieveid/insert_into_this_other_table/make_me_a_sandwich/and_dont_forget_coffee, in a few words do all the work, which - I HOPE ! - will be a faster and less stressfull method for the database.
Is this the good way to go ? Is this the best way to go ?
And if it is, what would be the skeleton of this mecha-query-of-death ? I cannot figure out how to insert-or-retrieve the phone ID in the same request of the insert-or-update parameters_list given this phone ID in the same request of a hundreds of other similar requests ?
Is this even possible ?
I hope you understand my nerves here have given up since a long time.
I would be happy and thankful for any help you can give to me.
Thank you.
What you want can be done with stored procedures. I will direct you at a couple of resources first -- since you appear to know a bit about MySQL already it's probably of benefit to you to learn how stored procedures work and code it yourself rather than have someone else just feed you the stored procedure.
http://code.tutsplus.com/articles/an-introduction-to-stored-procedures-in-mysql-5--net-17843
http://www.mysqltutorial.org/mysql-stored-procedure-tutorial.aspx
Related
I am new in forum, and need some help to do one functionality for my unity game. I am trying to save the progress of the player in one mysql database, like this:
userid level stars
29 1 2
29 2 1
45 1 3
50 1 2
50 2 3
50 3 1
29 3 3
so the script send the userid provided by the user registration in the begining of the game. and for each level he complete, the script send the number of the level, and the amount of stars collected in the level..
the problem and question is, how I configure this in the php script and mysql database to save the information only once? because if the player with the id 50 play the first level, will add a line with the information, but if the same player play the first level again and change the amount of stars, I dont want a new line, just update the stars amount.
I take a look in the INDEX, UNIQUE, PRIMARY, FULLTEXT, SPATIAL functions but dont figured out what is the correct combination and how to put in the php script, and take a look in other questions in the forum but nothing like this.
thanks for the help!
I recommend you use http://redis.io/ (in-memory data structure store, used as database, cache and message broker) to save progress in the games.
First you want an unique index on the combination (userid, level) and then you want to do an update if the combination exists and an insert otherwise.
For how to create the unique index please take a look at How do I specify unique constraint for multiple columns in MySQL?
For how to code the SQL query to do update/insert please take a look at SQL: If Exists Update Else Insert
The article above uses Microsoft SQL syntax. In PHP you can code this by issuing the query and then using mysql_affected_rows to see how many rows where affected. If 0 rows where affected then you issue the INSERT query from PHP.
in pseudo code you need to do something like this in SQL.
UPDATE $table set column=value WHERE id=$ID
Hi brayan actually the problems is that no one will write code for you, you have to do it yourself. I guess you are unaware with SQL i.e., you asked that
how I configure this in the php script and mysql database to save the
information only once? because if the player with the id 50 play the
first level, will add a line with the information, but if the same
player play the first level again and change the amount of stars, I
dont want a new line, just update the stars amount.
Anyhow You first required some basic understanding of SQL and PHP with Unity. I will recommend you this Guide Server_Side_Highscores of unityWiki it help you to make database and server logic intergartion with PHP.
Now for your Second important part of question.
You have to update user code after each level completion.Or you can simply ask to user about socre save.
Before inserting new record into the database you have to check that userId with level id alread exist or not. some thing like this
Select userid, level, stars
from youTableName
where userid = ?
and level = ?
if the above query return empty response then you simply need to add the record
INSERT INTO table_name (userid, level, stars)
VALUES (value1,value2,value3);
Otherwise you have to update that specific column.
I build a like system for a website and I'm front of a dilemma.
I have a table where all the items which can be liked are stored. Call it the "item table".
In order to preserve the speed of the server, do I have to :
add a column in the item table.
It means that I have to search (with a regex in my PHP) inside a string where all the ID of the users who have liked the item are registered, each time a user like an item. This in order verify if the user in question has (or not) already liked the item before. In this case, I show a different button on my html.
Problem > If I have (by chance) 3000 liked on an item, I fear the string to begin very big and heavy to regex each time ther is a like
on it...
add a specific new table (LikedBy) and record each like separately with the ID of the liker, the name of the item and the state of the like (liked or not).
Problem > In this case, I fear for the MySQL server with thousand of rows to analyze each time a new user like one popular item...
Server version: 5.5.36-cll-lve MySQL Community Server (GPL) by Atomicorp
Should I put the load on the PHP script or the MySql Database? What is the most performant (and scalable)?
If, for some reasons, my question does not make sens could anyone tell me the right way to do the trick?
thx.
You have to create another table call it likes_table containing id_user int, id_item int that's how it should be done, if you do like your proposed first solution your database won't be normalized and you'll face too many issues in the future.
To get count of like you just have to
SELECT COUNT(*) FROM likes_table WHERE id_item='id_item_you_are_looking_for';
To get who liked what:
SELECT id_item FROM likes_table WHERE id_user='id_user_you_are_looking_for';
No regex needed nothing, and your database is well normalized for data to be found easily. You can tell mysql to index id_user and id_item making them unique in likes_table this way all your queries will run much faster
With MySQL you can set the user ID and the item ID as a unique pair. This should improve performance by a lot.
Your table would have these 2 columns: item id, and user id. Every row would be a like.
I have recently written a survey application that has done it's job and all the data is gathered. Now i have to analyze the data and i'm having some time issues.
I have to find out how many people selected what option and display it all.
I'm using this query, which does do it's job:
SELECT COUNT(*)
FROM survey
WHERE users = ? AND table = ? AND col = ? AND row = ? AND selected = ?
GROUP BY users,table,col,row,selected
As evident by the "?" i'm using MySQLi (in php) to fetch the data when needed, but i fear this is causing it to be so slow.
The table consists of all the elements above (+ an unique ID) and all of them are integers.
To explain some of the fields:
Each survey was divided into 3 or 4 tables (sized from 2x3 to 5x5) with a 1 to 10 happiness grade to select form. (questions are on the right and top of the table, then you answer where the questions intersect)
users - age groups
table, row, col - explained above
selected - dooooh explained above
Now with the surveys complete and around 1 million entries in the table the query is getting very slow. Sometimes it takes like 3 minutes, sometimes (i guess) the time limit expires and you get no data at all. I also don't have access to the full database, just my empty "testing" one since the costumer is kinda paranoid :S (and his server seems to be a bit slow)
Now (after the initial essay) my questions are: I left indexing out intentionally because with a lot of data being written during the survey, it would be a bad idea. But since no new data is coming in at this point, would it make sense to index all the fields of a table? How much sense does it make to index integers that never go above 10? (as you can guess i haven't got a clue about indexes). Do i need the primary unique ID in this table? I
I read somewhere that indexing may help groups but only if you group by the first columns in a table (and since my ID is first and from my point of view useless can i remove it and gain anything by it?)
Is there another way to write my query that would basically do the same thing but in a shorter period of time?
Thanks for all your suggestions in advance!
Add an index on entries that you "GROUP BY" or do "WHERE". So that's ONE index incorporating users,table,col,row and selected in your case.
Some quick rules:
combine fields to have the WHERE first, and the GROUP BY elements last.
If you have other queries that only use part of it (e.g. users,table,col and selected) then leave the missing value (row, in this example) last.
Don't use too many indexes/indeces, as each will slow the table to updates marginally - so on really large system you need to balance queries with indexes.
Edit: do you need the GROUP BY user,col,row as these are used in the WHERE. If the WHERE has already filtered them out, you only need group by "selected".
I've got a problem that I just can't seem to find the answer to. I've developed a very small CRM-like application in PHP that's driven by MySQL. Users of this application can import new data to the database via an uploaded CSV file. One of the issues we're working to solve right now is duplicate, or more importantly, near duplicate records. For example, if I have the following:
Record A: [1, Bob, Jones, Atlanta, GA, 30327, (404) 555-1234]
and
Record B: [2, Bobby, Jones, Atlanta, GA, 30327, Bob's Shoe Store, (404) 555-1234]
I need a way to see that these are both similar, take the record with more information (in this case record B) and remove record A.
But here's where it gets even more complicated. This must be done upon importing new data, and a function I can execute to remove duplicates from the database at any time. I have been able to put something together in PHP that gets all duplicate rows from the MySQL table and matches them up by phone number, or by using implode() on all columns in the row and then using strlen() to decide the longest record.
There has got to be a better way of doing this, and one that is more accurate.
Do any of you have any brilliant suggestions that I may be able to implement or build on? It's obvious that when importing new data I'll need to open their CSV file into an array or temporary MySQL table, do the duplicate/similar search, then recompile the CSV file or add everything from the temporary table to the main table. I think. :)
I'm hoping that some of you can point out something that I may be missing that can scale somewhat decently and that's somewhat accurate. I'd rather present a list of duplicates we're 'unsure' about to a user that's 5 records long, not 5,000.
Thanks in advance!
Alex
If I were you I'd give a UNIQUE key to name, surname and phone number since in theory if all these three are equal then it means that it is a duplicate. I am thinking so because a phone number can have only one owner. Anyways, you should find a combination of 2-3 or maybe 4 columns and assign them a unique key. Once you have such a structure, run something like this:
// assuming that you have defined something like the following in your CREATE TABLE:
UNIQUE(phone, name, surname)
// then you should perform something like:
INSERT INTO your_table (phone, name, surname) VALUES ($val1, $val2, $val3)
ON DUPLICATE KEY UPDATE phone = IFNULL($val1, phone),
name = IFNULL($val2, name),
surname = IFNULL($val3, surname);
So basically, if the inserted value is a duplicate, this code will update the row, rather than inserting a new one. The IFNULL function performs a check to see whether the first expression is null or not. If it is null, then it picks the second expression, which in this case is the column value that already exists in your table. Hence, it will update your row with as much as information possible.
I don't think there're brilliant solutions. You need to determine priority of your data fields you can rely on for detecting similarity, for example phone, some kind of IDs, of some uniform address or official name.
You can save some cleaned up values (reduced to the same format like only digits in phones, concatenated full address) along with row which you would be able to use for similarity search when adding records.
Then you need to decide on data completeness in any case to update existing rows with more complete fields, or delete old and add new row.
Don't know any ready solutions for such a variable task and doubt they exist.
I need to generate a unique random code to store in a database with a user id.
What I'm trying to do is create a php script that first generates a random string of a given length, then checks a database to see if that string already exists, if so, generate a new random string.
The database will be organized by email address or some other field like customer_id. Each user can have say up to 5 devices associated with their account.
As a test I've created two MYSQL tables, one called users: email, firstname, lastname
the other called udevices. Udevices has 6 fields, one for the email address and 5 for the devices: email, dev1, dev2, dev3, dev4, dev5
all fields in both tables are VARCHAR
It occurs to me that another way to organize this is to have just two fields - email and device and then for each device just add another record to devices. Not sure which is most efficient.
So what i'm looking for is how to write a SELECT statement that will query the database for a given email address and a device string.
So, to boil the question down:
Can someone give me an example of a SELECT statement as described above? Is this even possible? Web searches on the topic bring up people talking about having to loop through each db record. Is that the only way, and if so, can someone give me an example of a PHP script that can loop through each record to check if a string already exists in a database?
You need a minimum of two tables, but most probably three if you need device descriptions, etc... I would go with three tables if I were you.
users: user_id | email | name | surname
devices: device_id | device_name | ...
user_devices: user_id | device_id
On users and devices the user_id and device_id must be the primary keys. On user_devices user_id and device_id must be the compound primary key.
Then the query to select all the devices of a user would be:
SELECT d.* FROM devices d
INNER JOIN user_devices ud
ON d.device_id = ud.device_id
AND ud.user_id = 123;
As far as the unique random code, you must tell us what its content will be (i.e. where will the uniqueness be based upon). If you have that, you can easily use one of the hashing functions such as md5(), etc... to generate the random string.
EDIT
If you do not need to verify the value of the random string, then you can generate one with the base_convert and microtime. The odds of duplicates are down to the microsecond. That is, if both visitors will request the code on that same microsecond they will get the same string, which is hardly ever the case, but still possible.
$string = base_convert(microtime(true), 10, 36);
it really depends on your final design, whether the udevices have static number of devices (as your current solution) or dynamic one (that 'another way' you stated). you don't need postprocessing via PHP, SQL alone can query it.
So what i'm looking for is how to write a SELECT statement that will query the database for a given email address and a device string
hey, isn't it too basic? read up your SQL book
Is that the only way, and if so, can someone give me an example of a PHP script that can loop through each record to check if a string already exists in a database?
what for? just SELECT and count the result. if it's > 0, then it exists. otherwise it doesn't.
A table with thing1, thing2, thing3 is clearly a repeating group, and should be normalized out. So your idea of having email/device is the right direction, however, using email address as the key is a bad idea. You are better off making a numeric auto_increment key for both tables, and having user_id be a foreign key in the device table that links them together, so that 1 user can have many devices associated with that user.
You can then query this table easily using select count(*) as countof from devices where device = '$devicename'. However, you can easily insure that your devices are unique by using something like:
$device = md5($email . uniqueid());