Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
What I Wish To Implement
My site does a nightly API data fetch, inserting 100,000+ new entries each night. To save space, each field name is in a seperate table with an allocated ID saving around 1,027 bytes per data set, 2.5675MB approx per night and just under a gigabyte over the course of a year, however this is set to increase.
For each user, a JSON file is requested containing the 112 entries to be added. Instead of checking my table for each name ID, I feel to save time, it would be best to create an array whereas the position in the array will be the ID, so lets use some random vegetable names;
Random List Of Vegetables
"Broccoli", "Brussels sprouts", "Cabbage", "Calabrese", "Carrots", "Cauliflower", "Celery", "Chard", "Collard greens", "Corn salad", "Endive", "Fiddleheads (young coiled fern leaves)", "Frisee", "Fennel"
When I create the insert via my PHP classes, I use the following;
$database->bind(':veg_name', VALUE);
Question
What would be the best method to quickly check what position $x is within the array?
As an alternative solution to matching the entries in PHP (which might at some point run into time and/or memory problems):
The general idea is to let the database to the work. It is already optimized (index structures) to match entries to one another.
So following your example, the database probably has a dimensional table for the field names fields:
ID | Name
---------------------------------
0 | "Broccoli"
1 | "Brussels sprouts"
2 | "Cabbage"
Then there is the "final" table facts, which has a structure like this:
User_ID | Field_ID | Timestamp
Now a new batch of entries should be inserted. For this, we first create a temporary table temp with the following format and insert all raw entries. The last column Field_ID will stay empty for now.
User_ID | Field_Name | Timestamp | Field_ID
In a next step we match each field name with its ID using a simple SQL query:
UPDATE `temp` t
SET Field_ID=(SELECT Field_ID FROM fields f WHERE f.Name=t.Field_Name)
So now the database has done our required mapping and we can issue another query to insert the rows into our fact table:
INSERT INTO facts
SELECT User_ID, Field_ID, Timestamp FROM temp WHERE Field_ID IS NOT NULL
A small side-effect here: All rows in our temp table, that could not be matched (we didn't have the field name in our fields table), are still available there. So we could write some logic to send an error report somewhere and have someone add the field names or otherwise fix the issue.
After we are done, we should remove or at least truncate the temp table to be ready for next nights iteration.
Small remark: The queries here are just examples. You could do the mapping and insertion into your facts table in one query, but then you'd lose the "unmatched" entries or have to redo the work.
Redoing the work might not be an issue now, but you said the number of entries will increase in the future, so this might become an issue.
If you're only doing 2.5 megs/night, that's almost nothing. If you gzipped that before dragging it across, it would reduce it a lot more.
Using array positions could get tricky if you're trying to use that to match something in some other table.
That being said, every array has a numeric index as well, so you can find out what that is at any point.
Try this and you'll see:
$array = array("Broccoli", "Brussels sprouts", "Cabbage", "Calabrese", "Carrots", "Cauliflower", "Celery", "Chard", "Collard greens", "Corn salad", "Endive", "Fiddleheads (young coiled fern leaves)", "Frisee", "Fennel");
var_dump(array_keys($array));
On the array, you can also do this:
$currentKey = array_search("carrot",$array);
That will return the key for a given variable. So if you're looping through an array, you can output the key(index) and go do something else with it.
Also, gzip is a form of compression that makes your data much smaller.
If you have a list of item, e.g. an array containing only strings that represent your values, you can use foreach with a key-value ($users as $index => $user) method instead of just a $users as $user like following :
$users = ["Broccoli", "Brussels sprouts", "Cabbage", "Calabrese", "Carrots", "Cauliflower", "Celery", "Chard", "Collard greens", "Corn salad", "Endive", "Fiddleheads (young coiled fern leaves)", "Frisee", "Fennel"];
foreach( $users as $index => $name ) {
echo "about to insert $name which is the #$index..." . PHP_EOL;
}
Which will echo :
about to insert Broccoli which is the #0...
about to insert Brussels sprouts which is the #1...
about to insert Cabbage which is the #2...
about to insert Calabrese which is the #3...
about to insert Carrots which is the #4...
about to insert Cauliflower which is the #5...
about to insert Celery which is the #6...
about to insert Chard which is the #7...
about to insert Collard greens which is the #8...
about to insert Corn salad which is the #9...
about to insert Endive which is the #10...
about to insert Fiddleheads (young coiled fern leaves) which is the #11...
about to insert Frisee which is the #12...
about to insert Fennel which is the #13...
Live-example available here : https://repl.it/Jpwk
Like #m13r asked, how would an index be useful in your case ?
Related
This question already has answers here:
mysql table with 40+ columns
(4 answers)
Closed 5 years ago.
Mysql tables.
phones
| ID | Name //5-6 columns
Specifications
| ID |phone_id| ram | camera | price |network // 50 columns approx
Now my specification table has 50 columns so far, so I need a suggestion how to handle this sort of situation ? Do I need to create some other tables to split specifications table or can I continue with this?
I need a better suggestion for better speed and performance.
Dealing with more than 50+ columns will not be best approach-
Data-Base operation like insert, update, select could take time.
Its became difficult task to handle the data when you are dealing
with 50+ columns in the table.
So I suggest do not continue with adding columns into the tables. mysql-table-with-40-columns could be usefull to solve your problems.
Perhaps you can have 1 main table with id, name, date and some more post specific stuff.
Then you could create 1 meta_table wich contains the id, main_id (this links to the post in your main table), meta_name (i.e.: price or color or dimensions), meta_value (i.e.: 125,95 or black or 150 x 50 x 8).
This way you can make as many column-like values stored into 1 table wich gets linked to 1 main table using an id.
This is the same concept as wordpress is using.
Otherwise you could also use something called serialized data/arrays into your main table, this is 1 column field with many array keys and their values.
This question already has answers here:
Split a comma-delimited string into an array?
(8 answers)
Closed 1 year ago.
Hi I am looking for someone to answer what is probably an easy question.
At the moment I have two tables "procedures" and "Job Roles", the idea is that I want to be able to create a child table that links which procedures apply to which job roles. At the moment this is done with two tables, by creating a record in the procedure table and then creating subsequent child records in a table called "procedures_jobtitles" which is just a one to many link table.
I am exploring the idea of having a field in the procedures table that is called "applies to", which would be a drop down box presented as check boxes, the idea is that when i create a procedure, on the add/edit page there is a separate tab where this field is displayed, this would display all job roles, so that the user could just tick the ones that apply.
My problem is by doing it this way the field value is stored as (1,2,5,10,11,12) where those numbers represent foreign keys to the job descriptions table. What I would like is to take the "applies to" field and turn it into an array and extract each value that is separated by a comma, so for the above example it would give me a result of:
1
2
5
10
11
12
I can then run a foreach query and insert these records into the link table by an event after saving. The idea behind this is if you have 1 or 2 job descriptions, manually adding to the child table isnt a problem, however when you have 20 or so job titles it becomes much more time consuming. Ive tried hitting google for this, but im afraid my basic understanding of arrays is limited so I dont know what to look for.
Any help using example fields ive given, would be much appreciated
Use explode()
e.g.
<?php
$mystring = "1,2,5,10,11,12";
$myarray = explode(",",$mystring);
//Output each element from $myarray:
foreach($myarray as $item)
{
echo $item."<br>";
//or do something else, e.g. insert these records into the link table
}
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I am currently trying to get values from tables I have in a webpage into a database using PHP. However, the order of the values inside each box is important, so I want the first box to be ranked 1 and the second box to be ranked 2 and so on... There is no limit to the number of items in each box. There may be as many as 100 or 0. Each item in each box is dragged from a bank of items into the table, which represents a topic. The tables are the output of the interface.
So example table:
Rank1
Rank2
Rank3
I've currently tried dumping the entire page once the user fills it in into a text file and parsing it from there but i'm looking for a more functional and practical way of doing it.
If I understand your goal correctly, you have two sections on your webpage. A word bank and a <table>. A user can drag and drop items from the word bank to a cell in your <table>. Next you want the order of the contents of the cell to be preserved when the data is saved to a database-table.
Given the description you have provided, it seems you may be new to using databases. It is not a problem but you will need to do some study on how databases work and how to use them.
Again, if I understand correctly, your <table> looks like the following:
| topic-1 | topic-2 | topic-3 |
-------------------------------
| ---A--- | ---J--- | ---V--- |
| ---B--- | ---J--- | ---X--- |
| ---C--- | NULL | ---Y--- |
| NULL- | NULL | ---Z--- |
To move this into a database you need to create a database, and some table, for example MyDataTable. Next that table may have three columns: id, contents, category.
Here is some info on Mysql Insert.
INSERT INTO MyDataTable(`id`, `contents`, `category`)
VALUES (1, A, Topic-1)
You will need to format that call to work correctly via php, but that is the general idea. In your php code you will need to iterate over the elements in the order you want, and you can use the id to refer to your order... Alternatively you could add another column to your database-table and have that refer to your order, which would allow you to use some other value, perhaps auto-increment, as the primary key.
I would recommend becoming comfortable with MySQL if you aren't already, before tackling your project's specific needs. Namely, INSERT, DELETE, UPDATE, SELECT and WHERE will almost certainly be necessary MySQL functions.
One resource I found very helpful in grasping MySQL is this book, SQL Antipatterns: Avoiding the Pitfalls of Database Programming .
Using the phpMyAdmin page can be very helpful in debugging SQL queries, once the query syntax is correct, you can translate it to php and incorporate your dynamic variables. Also if you store your sql-syntax in php as $sql, using var_dump($sql) can also be a great help when debugging sql syntax.
In programming, iterate over, typically refers to creating a for-loop to iterate, or step through 1-by-1, each of the items of an array or similar container.
$MyArray = array();
$MyArray = GetTableContents(); //You have to write this function.
for ($idx = 0; $idx < count($MyArray); $idx++)
{
// do my stuff, like, Insert into my DB-table.
}
In order for the code above to work, you need to fill MyArray with the contents from your <table>. In the example code above, the $idx, could be used as your id.
In addition to MySQL, it seems it would also be worth your time to learn more about php programming. Read simple tutorials, how-to's, and books. There are numerous resources on the internet about php. One such example is here.
I hope this is helpful.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
What's the best way of storing options in mySQL? As descriptive strings or integers that are associated with each string.
Let's say I have this question in my UI:
What's your favorite ice cream flavor?
Vanilla
Chocolate
Strawberry
Is it better to store those in the DB as 1, 2, 3 in an INT(1) field or as strings vanilla, chocolate, or strawberry, in a CHAR field? I know the INT field will be faster, but probably not drastically unless there's tens of thousands of rows.
If they're stored as strings then I wouldn't need to do any extra PHP code, whereas if they're stored as numbers, I'd have to define the value of 1 = vanilla, etc...
What's the general consensus on this?
The usual approach with relational databases is to make a new table called icecream_flavor or whatver. Then you can add new flavours at a later date, and your program can offer all the current flavour choices when it asks. You can store choices by table ID (ie an integer).
If paddy answer isn't an option then you should store the values as ENUM.
While ENUM is pretty equivalent to TINYINT(1).
ENUM is only the answer if the values you let the user choose are already pre-fixed, otherwise you would have to edit the table. But if you use ENUM MySQL has the engine optimized while inserting, and selecting from ENUMs. It's the obvious choice, for example (Male\Female).
Otherwise the answer to your question is TINYINYT(1), which is fastest then both CHAR and INT(1).
If you have a constant set of values and you're not going to relate that values with another data in your database (information related to each type of ice-cream) you can use MySQL's special field type called ENUM
If each user is identifiable by some unique key (say an email address) then you may find you do not need a numerical id.
Keeping the flavour as a flavour when YOU control the option (so you dont get Vanilla, vanilla vanila and so on) suggests to me that you store the real value (Vanilla).
This avoids a join and makes your database data meaningful when you browse it.
You can add an index to the flavour column of the database, so you can "show all users who prefer Vanilla" very cheaply.
(If you potentially have infinite options to store, then maybe you should be investigating using a nosql databases)
I would suggest that you keep the primary index as an int. The simplest reason for doing this is for what you don't currently know and to allow your data model to evolve.
Assuming your example, if you go with the actual word, and in six months you decide to also put in a size - single scoop, double scoop etc, you suddenly have a problem as your primary key that you thought was unique suddenly has multiple entries for each flavour. If on the other hand you go with a numeric format, you can easily have as many variants of Vanilla as you like and everything is happy.
In this case, I would also suggest keeping the actual primary key as a key and nothing more. Add in another column that then has the flavour, or create a lookup table that will store the actual properties of each item. If you keep a numeric format, but have ID 1 as Vanilla, you basically run into the same problem as before, keep the ID as the ID and your details seperate.
To keep data about your items, you can use something like this:
Master Table
ID Name
1 Ice Cream
Properties Table
ID Type
1 Flavour
2 Size
Property Table
ID PropID Detail
1 1 Vanilla
2 1 Strawberry
3 2 Single Scoop
4 2 Double Scoop
MasterToProperty
MasterID Property PropID
1 1 1
1 2 4
This basically gives you an unlimited number of options, and adding any thing you want (an entry for choc chips for example) is simply a matter of adding a few rows of data, not table changes.
Set up your database like this:
Questions
id, question_text
ex. (1, "What is your favorite ice cream?")
Options
id, question_id, option_text
ex. (1, 1, "Vanilla")
Responses
id, user_id, question_id, option_id
ex. (1, 421, 1, 1)
You should also throw some created and modified fields into each table for keeping track of changes.
I have a game. In the game, people make many choices out of 2 options.
The choice can be either right or wrong and I am storing the result of their run through the game (which can be a very large length) as a string with 1 for a right answer and 0 for wrong answers.
So for example, player 128937 will have stored in his run column the string 00010101010010001010111 as a varchar(5000).
Is there a better way I can store this information in MYSQL? (I am using PHP too if that can help)
I would create a new table (say it's called 'answers') with three columns:
question_id,user_id and answer (which will hold values of 0/1 )
every time the player answers a question you INSERT a new entry to this table.
This way it'll be easier to maintain the sum of right/wrong answers
Why not use a tinyint(1) for each option rather than using strings?
I would make multiple tables
choices
id
scenario (or other title)
options
id
choice_id
title (example: "go left" or "turn around and go home"
correct (0 or 1)
user_choices
user_id
option_id
choice_id (optional since choice_id is already in options table)