I have a question with im really unsure with.
First feel free to downvote me if its a must but i would really like to hear a more experienced developers opinion.
I am building a site where i would like to build similar functionality like google circles.
My logic would be this.
Every user will have circles attaced to them after signup.
example if the user will sign up
form filed and the following querys will be insierted to the database
**id | circle_name | user_id**
------------------------------------
1 | circle one | 1
------------------------------
2 | circle two | 1
------------------------------
3 | circle three | 1
Every circle will have a primary key
But this is what im unsure with, so after a time im a bit scared that the table will break, what im mean is if it will reach a number of id's it will actually stop generating more.
When you specifiy an int in the database the default value is 11, yes i know i can incrase or set it to the value what i want, but still giveing higher values is a good idea?
or is there any possibility to make a primary key auto increment to be unlimited?
thank you for the opinions and help outs
or is there any possibility to make a primary key auto increment to be unlimited?
You can use a BIGINT.
Strictly speaking it's not unlimited, but the range is so incredibly huge that you wouldn't be able to use up all the values even if you tried really hard.
Just run some maths and you ll get the answer yourself. If a length can store billions of values and you don't expect to have 1 million new registrations every week then getting to a point where it breaks would be "practically" tough, even if "theoretically" possible
Related
I suppose mariadb is working similarly to mysql, this is what I'm using, and I know there is a cache system.
My problem and what I don't understand, is that the pages that I refresh takes a long time to refresh, but the time is not constant at all. Details later.
On page A:
85% of the time, it takes ~7 seconds to execute.
10% of the time, it takes ~27 seconds.
5% of the time it takes under 1 second (when I refresh in very short intervals).
On page B:
80% of the time, it takes ~5 seconds.
Sometimes it's ~2.5 seconds.
Sometimes it's less than a second.
One time it has been >60 seconds, triggering an error.
My code is not changing, it's just observation and refreshing with F5.
Details:
I have a MyISAM table, that gets roughly 150k new rows ("insert") per day.
I am looking to query this table every minutes ("select").
The max rows it could have at a time might range between 50,000,000 and 4,750,000,000...
I'm using PHP to run the queries on the same server.
Structure I'm using currently:
CREATE TABLE `ticks` (
`primary` int(11) NOT NULL AUTO_INCREMENT,
`datetime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`pairs` text NOT NULL,
`price` decimal(18,8) NOT NULL,
`daily_volume` decimal(36,8) NOT NULL,
PRIMARY KEY (`primary`),
KEY `datetime` (`datetime`)
) ENGINE=MyISAM AUTO_INCREMENT=4007125 DEFAULT CHARSET=latin1
Data sample :
|primary | datetime | pairs | price | volume |
-------------------------------------------------------------------------------
|5810228 | 20/01/2018 21:34:02 | BTC_HUC | 0.00002617 | 6.08607929 |
|5810213 | 20/01/2018 21:34:02 | BTC_BELA | 0.00002733 | 8.83542600 |
|5810224 | 20/01/2018 21:34:02 | BTC_FLDC | 0.00000374 | 12.72654326 |
|5810234 | 20/01/2018 21:34:02 | BTC_NMC | 0.00037099 | 4.06446745 |
|5810219 | 20/01/2018 21:34:02 | BTC_CLAM | 0.00070798 | 13.65356478 |
|5810220 | 20/01/2018 21:34:02 | BTC_DASH | 0.07280004 | 423.88604591 |
|1706999 | 11/01/2018 17:09:01 | USDT_BTC | 13590.45341401 | 398959280.2620621|
I have created an index ("normal" index) on datetime.
The query on page A that takes 7 seconds to run with pdo, but ~0.0007 in phpmyadmin :
SELECT DISTINCT(pairs)
FROM ticks
Every heavy computations after this first query takes ~0.5 seconds total most of the time since I indexed datetime.
However, it sometimes takes between 25 and 35 times longer to run for unknown reasons. This is the query that is used (a loop runs it 100 times) :
SELECT datetime, price
FROM ticks
WHERE datetime <= DATE_SUB(NOW(),INTERVAL 1 MINUTE)
AND pairs = \''.$data['pairs'].'\'
ORDER BY datetime DESC
LIMIT 1
I'm not going further into explaining page B because this page is less critical for me and I'm comfortable with the avg execution time related to the number of operations made on this page. My only interrogation is the wide range of execution times that can occur here too.
Questions:
1-How can the execution time differences be so wide, how can I have my pages running in under 1 sec, as it happens sometimes ? My sql queries are extremely simple and fast on the database alone. I believe the db and the php server is located on the same machine.
In particular, I'm wondering why a query would run 10,000 slower with pdo than with phpmyadmin. 7/0.0007 being 10k, there has to be a huge problem here.
Indexing pairs is not changing anything.
2-Have you seen anything incorrect in what I explained that could lead to a fix and improvement of performances? Do you have particular advises to have increased performance in the presented case? For instance, I've been wondering if MyISAM was efficient in my case (I believe so).
There is essentially no reason to use MyISAM any more, especially for performance.
7 seconds is terrible for a page load. How much of that is MySQL actions? Add some timers in the code. This will find out which query is the slowest and let's improve it. (I would guess that one unnecessarily slow query is at the root of your problem.)
"~0.0007" smells like the Query Cache kicked in and it did not really execute the query. I ignore that.
With MyISAM, INSERTs block SELECTs. That could explain the troubles during the insert part of the day.
The table is confusing -- you have a TIMESTAMP (resolution to second), yet there is a "daily_volume" which sounds like a resolution to the "day".
I see TEXT. How long are the rows? If less than 255, use VARCHAR, not TEXT. That would allow you to add INDEX(pairs), which allowSELECT DISTINCT(pairs) FROM ticks to run a lot faster.
But, instead of that index, add INDEX(pairs, datetime) in order to make the second SELECT run much faster.
Shrinking the table size will help some in speed. (By some, I mean anywhere between 10% and 10x, depending on a lot of factors.)
Your decimal sizes are excessive. Find the worst (probably BRKA) and shrink the m,n of DECIMAL(m,n). Currently you are using 9 and 15 bytes for those two columns. You might consider FLOAT (4 bytes, ~7 significant digits) or DOUBLE (8 bytes, ~16 digits).
See my notes on converting to InnoDB . Be aware that the disk footprint might double or triple. (Yes this is an advantage of MyISAM.)
Consider whether some other column (or combination of columns) is unique. If you have such, jetison the column primary and make that column(s) the PRIMARY KEY. If it happens to be (pairs, datetime), then that will give a further performance boost to some queries.
"Indexing pairs is not changing anything." -- Since you can't index a TEXT column without using "prefixing" and prefixing is virtually useless, I am not surprised.
Could you show me a sample of the data? I am not familiar with what a "pair" is.
An index starting with TIMESTAMP or DATETIME is rarely useful; get rid of it unless you have another query that benefits from it.
As for the Query Cache -- size should be no more than 50M. Does the data not change for 23 hours of the day, then there is a flurry of inserts? This would be a good case for using the QC. (Most production servers are better off turning it OFF.) Going above 50M may slow down performance.
After you have addressed most of my suggestions, some other issues may bubble to the surface. That is, I expect you to come back with another Question to finish improving the performance for your app.
How can the execution time differences be so wide, how can I have my pages running in under 1 sec, as it happens sometimes ? My sql queries are extremely simple and fast on the database alone.
It's impossible to answer this question with any degree of certainty without analyzing your platform, monitoring the performance each component, reviewing the code and all the queries, etc. This is way beyond the scope of SO.
What can be said is:
It's unlikely that it has anything to do with PDO itself (or PHPMyAdmin for that matter)
It's typical of a concurrency problem - that is unless you have a server and a database dedicated to rendering "page A" only, other requests and queries happening at the same time can impact performances
MyISAM is notoriously bad at handling a large volume on insert because it uses table locking (in short, it locks all the table every time you make an insert). InnoDB use row based locking which would very probably be much more efficient with 150k writes a day. To quote the MySQL Documentation:
Table locking enables many sessions to read from a table at the same time, but if a session wants to write to a table, it must first get exclusive access, meaning it might have to wait for other sessions to finish with the table first. During the update, all other sessions that want to access this particular table must wait until the update is done.
I searched in the internet for an answer to select every columns that matches regex pattern. I didn't find one, or maybe I did, but I didin't understand it, because I'm new to DataBases. So here's the sql I was trying to run:
UPDATE `bartosz` SET 'd%%-%%-15'=1
(I know it's bad)
I have columns like:
ID | d1-1-15 | d2-1-15 | d3-1-15 | d4-1-15 ... (for 5 years, every month, and day)
So is there a way to select all columns from 2015?
I know i can loop it in php so the sql would look like:
UPDATE `bartosz` SET 'd1-1-15'=1, 'd1-1-15'=1, 'd3-1-15'=1 [...]
But it would be really long.
Strongly consider changing your approach. It may be technically possible to have a table with 2000 columns, but you are not using MySQL in a way that gets the most out of the available features such as DATE handling. The below table structure will give better flexibility and scaling in most use cases.
Look into tables with key=>value attributes.
id employee date units
1 james 2015-01-01 2
2 bob 2015-01-01 3
3 james 2015-01-02 6
4 bob 2015-01-02 4
With the above it is possible to write queries without needing to insert hundreds of column names. It will also easily scale beyond 5 years without needing to ALTER the table. Use the DATE column type so you can easily query by date ranges. Also learn how to use INDEXes so you can put a UNIQUE index on the employee and date fields to prevent duplication.
I have a PHP website which takes customer applications. Each application is given an ID number which is incremental.
Recently the site reached application id number 9999 but instead of continuing on to 10000 it reverted back to 0001
Any ideas why this happened - perhaps some kind of php or mysql setting, range or limit?!
Thanks in advance
it is likely the size of the int allowed for the incrementing primary key.
please do this in table structure for id int(11)
in length give 11
yes there was some forming applied in the php to limit to 4 digits (0000) so when it reached 10000 it was cutting off the first 1. This was causing it to appear as 0001 etc in the backend and other places...
As a consequence it was also sending out the cut version of the id nmber in emails to applicants and causing all sorts of mess lol!
Took a while to find the right file because the original coder is not available!
Thanks to all.
I am starting to think about my new project and I've found a couple of speed issues, so I hope you can help me with selecting a good and elegant way to code it.
Each user has in the database records of "places" he has visited. Each place has "schools" - a number of schools in this particular place. Each school has classes. Each class may end its "learning year" at different times, so it's number should increment if date is >= end of learning year.
So we have such a database:
"places" table:
place | user_id |
-----------------
1 | 4 |
2 | 4 |
User no 4 visited place no 1 and 2
"schools" table:
school | place |
----------------
5 | 2 |
6 | 2 |
Place 2 has two schools - with id 5 and 6.
"class" table:
class | school | end_learning | class_number
---------------------------------------------
20 | 5 | 01.01.2013 | 2
21 | 5 | 03.01.2013 | 3
22 | 5 | 05.01.2013 | 4
School 5 has 3 classes with ids 20, 21, 22. If date is greater than 01.01.2013, the class number of class 20 should be incremented to 3 and end learning date changed to 01.01.2014. And so on.
And now we got into the problem - if there is 1000 places, each with 100 schools, each with 10 classes we got 1000000 records. It's a lot. Because all I have presented is just a simple example I have to consider updating whole database every time user refreshes the page so I'm afraid it might be laggy on that amount of records.
I also can serialize class into one field in school table:
school | place | classes
-------------------------------------------------------------------------
5 | 2 | serialized class 20, 21, 22 with end_learning field and class number
6 | 2 | other serialized classes from school 6
In that case I get 10 times less records but each time I have to deserialize data, check dates and if it's less than now alter it, serialize and save to database. The second problem is that I have to select all records from db to manipulate them not only all those need to be altered.
I am also thinking about having two databases: One with records that might need change in further future, and second that might need change in next 24hrs (near future). Every 24hrs all the classes which end learning in next 24 hrs are moved to "near future" db so every refresh of the page works on thousands of records, not hundreds of thousands or millions. Instead of that it works on millions of records (further future) to create "near future" table only once per day.
What do you think about all those database schemas? Maybe you have a better idea?
I don't quite understand the business logic or data model you outline - but I will assume you have thought this through.
Firstly, RDBMS solutions like MySQL are really, really good at managing large numbers of records, as long as the data you are working with is relational. As far as I can tell, you will be searching across many records, but only updating a few (a user will only be enrolled in a limited number of classes); I don't see this as a huge problem.
Secondly, it's nearly always better to go with the "standard" relational model until you can prove it doesn't meet your performance needs than to go for "exotic" solutions at the start off (I class your serialization and partitioning solution as "exotic" for the purpose of this answer). A lot of time and energy has gone into optimizing performance of SQL; if there were a simple alternative, it would be part of the standard solution. There are, of course, points at which the standard relational model doesn't scale (Facebook-size traffic, for instance), or business domains where the relational model doesn't really fit (documents, graphs). However, all the alternatives have benefits and drawbacks just like "standard" MySQL.
Thirdly, the best way to deal with possible performance issues is, well, to deal with them. In code. Build a test rig, create a schema according to the relational model, populate it with test data (e.g. using DbMonster), throw some load at it (e.g. using JMeter) and tune your schema and queries to prove your situation doesn't fit the standard solution. Only go for something exotic if you really can prove that you can't play nice with standard, relational database stuff.
Site.
This code is not optimized nor is it the best method. If you have any ideas to improve anything, let me know.
Please visit the site to get an idea of the data.
I have used Dan G. Switzer, II's calculation plugin [adding .sum() .max() .min() .avg()]
The validation requirement i'd like to have is to make sure nothing conflicts with that user's already determined range. Also, that there are no gaps in the range.
For example give
Brian 40 50 1200
Brian 50 70 1200
I dont want to the user to be able to set the first 50 to 39 because 39 would be smaller than 40. I dont want them to let 50 be set to anything higher than 50 because it would overlap the next range.
Any good ideas? perhaps actually running through all the values and then making a real range and then on BLur have it check to make sure no range is actually overlapped or gapped.
Each unique input is defined as id=NAME, so if I wanted to reference all the inputs of Brian, I could use $("input[id='Brian']").each() or if I wanted to reference all the START inputs of Brian, I can use $("input[id='Brian'][name='start[]'").each()
Edit:
One thing to note is that the page is PHP, and PHP is ran to populate the inputs via a CSV file. It will always start with correct data, and PHP can be used to help create ranges.
because of this I was thinking of just disabling the START field, because it will always populate the next range. However, I will be adding the ability to delete rules, so that can get messy if they are limited in what they can do.
One of the problems I see is that the Name fields is editable. If the field is changed, all the overlapping has to be recalculated. Not only is that is a performance issue, it is also a usability problem: what if one wants to change a name and the script, seeing discrepancies, forbids it?
A solution would be to change the table to one resembling the diagram below:
[ Add Employee ]
| Name | Start | End | Wages |
==========================================
| | 0 | 50 | 100 |
| Brian | 50 | 70 | 150 |
| | 100 | 150 | 200 |
| [ Add Rule ] |
------------------------------------------
| Another | 0 | ... | etc |
The [ Add Employee ] button would ask for a name and add a cell in the first column, and a line in the next three: with a default Start value of 0. The user can then enter data on the next fields, and can [ Add Rules ] as wanted. A good restriction could be to lock the values of the Start column and set them to the End value of the previous line.
$('#Brian .LineN input.start').val(
$('#Brian .LineN').prev().find('input.end').val()
);
Then adding a comparison function to check if the End value is lower than the Start value is trivial to implement.
Deleting a rule could simply follow the same procedure as for adding a line: the line below the one deleted (if any) would set its Start value as the now previous line's End.
The real difficulty would be to insert lines at arbitrary points. I'm not going to think about that one, though.
EDIT:
If a rewrite is out of the question, then just disabling the start should be enough. However, IMHO I would rather (re)write something with no caveats than spend time later on numerous bugs and feature requests.