I am newbie in cakephp, and I am confused about SHOW FULL COLUMN query
I've set debug leel into 2, and I clicked on one page, its show 88 queries (even the data is NULL)
Here the queries:
SHOW FULL COLUMNS FROM item_purchase_returns 10 10 2
SHOW FULL COLUMNS FROM purchases 20 20 2
SHOW FULL COLUMNS FROM outcomes 13 13 1
SHOW FULL COLUMNS FROM transaction_categories 7 7 96
SHOW FULL COLUMNS FROM incomes 16 16 2
SHOW FULL COLUMNS FROM last_receivables 14 14 1
SHOW FULL COLUMNS FROM last_funds 7 7 1
...
etc
My question is why does cakePHP do this?
Does those queries are usefull?
Is cakePHP calls those queries on each page?
Thanks!
CakePHP calls those queries in order to build it's internal model of your database tables. It does call them quite often in development mode (the cache is flushed every 10 seconds) in case you make changes to your database, but if the debug level is set to production (0), it will assume you aren't making any database changes and only flush the cache every 99 years (essentially never). Thus, if you wish to make changes to the database of an app in production, you either have to delete your cached models in /app/tmp/ or temporarily change to development mode.
Related
I'm using tokens for how many messages a user can send (1 message requires 1 token). At the moment I've just got it subtracting the value from an overall value to check if the user has tokens remaining and that's working fine.
I'm trying to change it so that it shows which bundle is active, so I need to check if the user doesn't have enough tokens remaining in the active bundle change to the upcoming_bundle.
Example:
Stored User Data:
Table Name: Tokens
First Record
id: 1
user_id: 5
bundle_type: small
value: 10
value_remaining: 4
state: active_bundle
Second Record
id: 2
user_id: 5
bundle_type: large
value: 100
value_remaining: 100
state: Upcoming_bundle
User sends 10 messages (10 tokens)
Only 4 remaining tokens in first record. Use 4 remaining tokens and leave
6 tokens
Then subtract the 6 tokens from second record which is now active so that will leave 94 remaining tokens.
Should I have a check to database every time the message is sent and update the database to subtract 1 token at a time, then when the remaining_value hits 0 change active_bundle to inactive and upcoming_bundle to active?
If this is your data model then I would fetch all active & upcoming bundles and then do the logic in php, e.g. subtract remaining tokens, change status, etc and then update them as a transaction.
If you are flexible on how the data is structured, I would rather have some kind of transaction log, from which I can read each action, i.e. whether a bundle was added or a token was used with a timestamp. For example like this:
id | user | change | comment | timestamp
1 | 1 | 10 | bought small bundle | 2016-09-06 09:30:00
2 | 1 | -1 | sent message | 2016-09-06 10:56:00
3 | 2 | -3 | sent multi-message | 2016-09-06 10:57:00
Where id is the transaction id, user the user id, change is the number of tokens added (by adding a bundle) or used (by sending one or many messages) and comment a message describing the action. When you want to find out how many tokens there are left you can just do a search for that user and check their SUM(change) instead of weird searches for active/upcoming bundles. Obviously this can be more or less elaborate depending on your needs.
This does not take into account your actual domain! There are more approaches each having their drawbacks. For example my approach might have problems wen the transaction_log-table gets large because of number of users and increased activity, although it is very unlikely (I have seen mysql perform well with a few million records in a similar log table). The important part is: You should figure out what is important to your use case and build a solution around the requirements.
What I would do is, I would subtract it one at a time, not only this is safer, but also a lot easier.
I have 2 million rows in a myqsl DB which have multiple columns of contacts as phone_1, phone_2 upto phone_10.
These phone no. may or may not duplicate.
I intend to group them together..as
ID Contact_1 Contact_2 Contact_3
P1 1 2 3
P2 5 6 7
P3 2 8 9
result should be:
ID Contact_1 Contact_2 Contact_3 Group
P1 1 2 3 1
P2 5 6 7 2
P3 2 8 9 1
P3 11 12 13 3
P3 7 21 22 2
Now where should I do the processing part ...PHP/Python or mysql.
i.e. select the entire data in php script and create an arrray and process tha array and then use insert query.
OR
select the entire data in php script and then use UPDATE(with a logic to create groups) query.
??
I have group field in DB table.
It depends, I'd say mostly on how comfortable you are with each language. I would probably do this with PHP, but that's the language I know best.
You can certainly do it purely with MySql, and the operation would probably run faster, but it might be easier to debug and test each step in PHP or Python.
Regardless, I'd recommend first creating a data subset, maybe 1000 or 2000 rows from the table and running everything against that until you're happy with the results. It'll be much much faster and you'll see mistakes sooner.
Also, I'd avoid running anything you're worried about being slow on a production server.
A client wants to be able to import a fairly hefty spreadsheet of data (to my eyes, at least) on a monthly basis. At the moment it consists of around 200 rows, and 41 columns.
The first two columns are identifiers (an ID and a human-readable name for a location); the next 39 columns are all numerical values. These numbers are effectively 'ratings' that are split into 39 categories - however the client has just hinted that they may want to expand these categories in the future.
My initial thoughts were to create a database structure that imitates the spreadsheet (so, 41 columns, essentially) - however with the potential for the client to want to add to it later, I don't know if that's the best method to approach this with.
I'm not a classically-trained coder; I've just picked things up as I've needed to, so the more straightforward the solution, the better. I have considered perhaps serializing an associative array on a per-location basis:
----------------------------------------------------------
ID Name Data Date
----------------------------------------------------------
1 Newport {niceness:1, staff:6.5, etc.} Mar 13
2 Stobart {niceness:7, staff:3.1, etc.} Mar 13
----------------------------------------------------------
...but I don't know if that's the best approach, or if it will necessarily work the way I envisage. The idea being that, if March '13 only has 39 categories, but July '13 has 42 categories, then the database structure won't have to be affected; just a different associative array stored.
This kind of thing would also potentially(?) remove the possibility for column mismatch errors when importing the table - for example, if they suddenly decide to drop a category without warning, then try importing an spreadsheet that's a column short.
Things of note:
I am developing this is a Wordpress plugin, so the database table structure is written in there and would be fairly straightforward to update, thanks to WP's dbDelta function.
I'm also looking to use Matt Kruse's PHP ExcelReader, which in my initial tests appears to be excellent.
I would split tables into names, categories and a link-table called *names_categories* (or similar)
table Names
----------------------------------------------------------
ID Name Date
----------------------------------------------------------
1 Newport Mar 13
2 Stobart Mar 13
table Categories
----------------------------------------------------------
ID Category
----------------------------------------------------------
1 niceness
2 staff
table names_categories (link-table)
----------------------------------------------------------
id_name id_category value
----------------------------------------------------------
1 1 1
1 2 6.5
2 1 7
2 2 3.1
This would give you more flexibility. Even if your solution works, it's not that flexible and could be hard to do maintenance of the database in future.
I am starting to think about my new project and I've found a couple of speed issues, so I hope you can help me with selecting a good and elegant way to code it.
Each user has in the database records of "places" he has visited. Each place has "schools" - a number of schools in this particular place. Each school has classes. Each class may end its "learning year" at different times, so it's number should increment if date is >= end of learning year.
So we have such a database:
"places" table:
place | user_id |
-----------------
1 | 4 |
2 | 4 |
User no 4 visited place no 1 and 2
"schools" table:
school | place |
----------------
5 | 2 |
6 | 2 |
Place 2 has two schools - with id 5 and 6.
"class" table:
class | school | end_learning | class_number
---------------------------------------------
20 | 5 | 01.01.2013 | 2
21 | 5 | 03.01.2013 | 3
22 | 5 | 05.01.2013 | 4
School 5 has 3 classes with ids 20, 21, 22. If date is greater than 01.01.2013, the class number of class 20 should be incremented to 3 and end learning date changed to 01.01.2014. And so on.
And now we got into the problem - if there is 1000 places, each with 100 schools, each with 10 classes we got 1000000 records. It's a lot. Because all I have presented is just a simple example I have to consider updating whole database every time user refreshes the page so I'm afraid it might be laggy on that amount of records.
I also can serialize class into one field in school table:
school | place | classes
-------------------------------------------------------------------------
5 | 2 | serialized class 20, 21, 22 with end_learning field and class number
6 | 2 | other serialized classes from school 6
In that case I get 10 times less records but each time I have to deserialize data, check dates and if it's less than now alter it, serialize and save to database. The second problem is that I have to select all records from db to manipulate them not only all those need to be altered.
I am also thinking about having two databases: One with records that might need change in further future, and second that might need change in next 24hrs (near future). Every 24hrs all the classes which end learning in next 24 hrs are moved to "near future" db so every refresh of the page works on thousands of records, not hundreds of thousands or millions. Instead of that it works on millions of records (further future) to create "near future" table only once per day.
What do you think about all those database schemas? Maybe you have a better idea?
I don't quite understand the business logic or data model you outline - but I will assume you have thought this through.
Firstly, RDBMS solutions like MySQL are really, really good at managing large numbers of records, as long as the data you are working with is relational. As far as I can tell, you will be searching across many records, but only updating a few (a user will only be enrolled in a limited number of classes); I don't see this as a huge problem.
Secondly, it's nearly always better to go with the "standard" relational model until you can prove it doesn't meet your performance needs than to go for "exotic" solutions at the start off (I class your serialization and partitioning solution as "exotic" for the purpose of this answer). A lot of time and energy has gone into optimizing performance of SQL; if there were a simple alternative, it would be part of the standard solution. There are, of course, points at which the standard relational model doesn't scale (Facebook-size traffic, for instance), or business domains where the relational model doesn't really fit (documents, graphs). However, all the alternatives have benefits and drawbacks just like "standard" MySQL.
Thirdly, the best way to deal with possible performance issues is, well, to deal with them. In code. Build a test rig, create a schema according to the relational model, populate it with test data (e.g. using DbMonster), throw some load at it (e.g. using JMeter) and tune your schema and queries to prove your situation doesn't fit the standard solution. Only go for something exotic if you really can prove that you can't play nice with standard, relational database stuff.
Since it looks like recursive queries aren't possible in MySQL, I am wondering if there is a solution to get the same information that also limits the number of queries I make to the database. In my case I have what amounts to a tree and given a node, I make a path back to the root and save the name of the nodes as I go. Given a table like this:
id | parent
-------------
1 |
2 | 1
3 | 1
4 | 2
5 | 2
6 | 5
I want to select all ids on the path from 6 back to 1 (6,5,2,1). Since the total length of the path is unknown I would assume that the only way to do this is taking the results from one query and build a new query until I am back at the root. Then again it has been a couple years since I last used MySQL so it wouldn't surprise me if I am a little out of touch. Any help would be appreciated.
Since it looks like recursive queries aren't possible in mySQL
mySQL doesn't support the 'CONNECT BY' operator, true - but you can implement recursive procedures/functions using mysql and return result sets from them.