Not sure how to word this but my problem is that my fields will not update properly. I have a page set up where the user can update things like job listings, events, etc. The problem is that some of the descriptions are 3 or more paragraphs long and when the form is processed, it doesn't update the database correctly. This also happens when the user makes a new item.
Is there a limit on how much text can be loaded at once?
Here is the code for updating that I am using:
mysql_query("UPDATE tbl_workers_club SET eventdate='".$newDate."', theme='".$theme."', text='".$text."', contactperson='".$contactperson."', contactphone='".$phone."', dateentered='".$dateentered."' WHERE specialID='".$id."' ");
Here is the code for new items:
mysql_query("INSERT INTO tbl_workers_club (eventdate, theme, text, contactperson, contactphone, dateentered)
VALUES ('$newDate', '$theme', '$text', '$contactperson', '$contactphone', '$dateentered')");
I did not see another question like this so if you know of one let me know.
Saying what errors you're getting (or no errors) would greatly help diagnosing this problem. Is the content merely "cut off" in the database? Or is MySQL throwing errors back to PHP? However, here are some ideas...
mysql_real_escape_string() is a good place to start. Any input with a single apostrophe will break your code. Read up on cleansing user input for SQL, there are tons of resources on the web. It's absolutely not optional. Also look into parameterized queries / prepared statements, or one of the many ORM libraries/frameworks out there.
It's also possible that your "text" column is not large enough to hold those inputs, if you're only having problems for very long entries. VARCHAR(X) can only hold X characters. Use something like MEDIUMTEXT instead.
Use POST rather than GET; GET has limits on the size of values sent via the URL. POST is the conventional method when sending requests that create/alter data.
It's hard to say what the problem is without more info on what errors you're getting.
take a look to the setup of MySQL max_allowed_packet A communication packet is a single SQL statement sent to the MySQL server, a single row that is sent to the client, or a binary log event sent from a master replication server to a slave.
mysql> select * from information_schema.global_variables where variable_name like 'max_allow%';
+--------------------+----------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+--------------------+----------------+
| MAX_ALLOWED_PACKET | 1048576 |
+--------------------+----------------+
1 row in set (0.00 sec)
Related
I have a very large database table (more than 700k records) that I need to export to a .csv file. Before exporting it, I need to check some options (provided by the user via GUI) and filter the records. Unfortunately this filtering action cannot be achieved via SQL code (for example, a column contains serialized data, so I need to unserialize and then check if the record "passes" the filtering rules.
Doing all records at once leads to memory limit issues, so I decided to break the process in chunks of 50k records. So instead of loading 700k records at once, I'm loading 50k records, apply filters, save to the .csv file, then load other 50k records and go on (until it reaches the 700k records). In this way I'm avoiding the memory issue, but it takes around 3 minutes (This time will increase if the number of records increase).
Is there any other way of doing this process (better in terms of time) without changing the database structure?
Thanks in advance!
The best thing one can do is to get PHP out of the mix as much as possible. Always the case for loading CSV, or exporting it.
In the below, I have a 26 Million row student table. I will export 200K rows of it. Granted, the column count is small in the student table. Mostly for testing other things I do with campus info for students. But you will get the idea I hope. The issue will be how long it takes for your:
... and then check if the record "passes" the filtering rules.
which naturally could occur via the db engine in theory without PHP. Without PHP should be the mantra. But that is yet to be determined. The point is, get PHP processing out of the equation. PHP is many things. An adequate partner in DB processing it is not.
select count(*) from students;
-- 26.2 million
select * from students limit 1;
+----+-------+-------+
| id | thing | camId |
+----+-------+-------+
| 1 | 1 | 14 |
+----+-------+-------+
drop table if exists xOnesToExport;
create table xOnesToExport
( id int not null
);
insert xOnesToExport (id) select id from students where id>1000000 limit 200000;
-- 200K rows, 5.1 seconds
alter table xOnesToExport ADD PRIMARY KEY(id);
-- 4.2 seconds
SELECT s.id,s.thing,s.camId INTO OUTFILE 'outStudents_20160720_0100.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
FROM students s
join xOnesToExport x
on x.id=s.id;
-- 1.1 seconds
The above 1AM timestamped file with 200K rows was exported as a CSV via the join. It took 1 second.
LOAD DATA INFILE and SELECT INTO OUTFILE are companion functions that, for one one thing, cannot be beat for speed short of raw table moves. Secondly, people rarely seem to use the latter. They are flexible too if one looks into all they can do with use cases and tricks.
For Linux, use LINES TERMINATED BY '\n' ... I am on a Windows machine at the moment with the code blocks above. The only differences tend to be with paths to the file, and the line terminator.
Unless you tell it to do otherwise, php slurps your entire result set at once into RAM. It's called a buffered query. It doesn't work when your result set contains more than a few hundred rows, as you have discovered.
php's designers made it use buffered queries to make life simpler for web site developers who need to read a few rows of data and display them.
You need an unbuffered query to do what you're doing. Your php program will read and process one row at a time. But be careful to make your program read all the rows of that unbuffered result set; you can really foul things up if you leave a partial result set dangling in limbo between MySQL and your php program.
You didn't say whether you're using mysqli or PDO. Both of them offer mode settings to make your queries unbuffered. If you're using the old-skool mysql_ interface, you're probably out of luck.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Today I was working on my website and I asked myself a simple question.
Does storing an array with all informations is better than saving those one in different fields?
For example if I store a word, a password and a number in one field on the database in this way
+-------------+----------------------------------------------------------------+
| Field | Value |
+-------------+----------------------------------------------------------------+
| all | ["test","fa26be19de6bff93f70bc2308434e4a440bbad02","25468684888"] |
+-------------+----------------------------------------------------------------+
Is it better than saving it in this way?
+-------------+------------------------------------------+
| Field | Value |
+-------------+------------------------------------------+
| word | test |
| password | fa26be19de6bff93f70bc2308434e4a440bbad02 |
| number | 25468684888 |
+-------------+------------------------------------------+
I think that the first method is faster than the last one because you need only to SELECT one field and not three or more. What do you think about it?
The second method. By far.
You should never put more than one piece of data into a single column.
A single row of data shuld contain all the information you need:
id name password
1 Fluff itsASecret
2 Flupp Ohnoes
Basically, it has to do with updates, selects, searches and pretty much everything that databases do. They are made to do it on single columns, not little bits of data inside a string.
Taking your example, how do you update the password? How do you put an index on the user ID?
What if you also had a bit of data called "NumberOfVotes" If you had it all in one column in a pseudo-array, how do you get a tally of all the votes cast by all users? Would you REALLY want to pull each entry out into PHP, explode it out, add it to the running total and THEN display how many votes have been cast? What if you had a million users?
If you store everything in a ingle column, you could do a tally really easily like this:
select
sum(NumberOfVotes)
from
yourTableName
Edit (Reply to faster query):
Absolutely not, the time it takes to compelte a query will come down to two things:
1) Time it takes to execute the query
2) Time it takes to return all the data.
In this case, the time it takes to return the data will be the same, after all, the database is returning the same amount of bytes. However, with tables that are properly set up, just FINDING the right data will be faster by orders of magnitue.
As an example of how difficult it would be to simply USE a table that has the various bits of information all mumbled together, try to write a query to update the "number" value in the row that starts with the word "test".
Having said that, there are possibly some potential cases where it can in fact be okay to store multiple "fields" of data in one column. I once saw (and copied) an exceptionally interesting permissions system for users that stored the various permissions in binary and each digit in the number equated to being allowed/not being allowed to perform a certain type of action. That was however one interesting example - and is pretty much what I would call an exception that proves the rule :)
I think that the first method is faster
is your main problem actually. You are comparing solutions from only "is it faster" point of view. While you have no measure to tell if there is any difference at all. Or, if even there is, if such a difference does matter at all. So, the only your reason is a false one. While you completely overlook indeed important, essential reasons like proper database design.
Saving in separate fields is a lot more flexible as you are then able to easily search/manipulate data using SQL queries, whereas if they were in an array you would frequently find yourself needing to parse data outside SQL. Consider the following example:
+-------------+----------------------------------------------------------------+
| Field | Value |
+-------------+----------------------------------------------------------------+
| all | ["1","fa26be19de6bff93f70bc2308434e4a440bbad02","25468684888"] |
+-------------+----------------------------------------------------------------+
Using the above table, you need to find the number field for the user with id 1, however there is nothing to search for, you can't simply to a query for the value 1 somewhere in the all field, as that would find every instance of the number 1!
You'll also encounter this problem when changing data in your DB, as you'll have to get the current array, parse it, change the value, then reinsert it.
Also you'll need to put some form of ID as a field to act as a primary key.
However with separate fields for each value, it's fairly simple:
+-------------+------------------------------------------+
| Field | Value |
+-------------+------------------------------------------+
| id | 1 |
| password | fa26be19de6bff93f70bc2308434e4a440bbad02 |
| number | 25468684888 |
+-------------+------------------------------------------+
SELECT `number` FROM mytable WHERE id = 1
The second option is better because its more readable and maintainable.
If someone who didnt write the code has to maintain it, the first option is terrible.
If you ever need to change a field, or add a field, likewise, the first option is a nightmare.
The second option requires much less work.
Keep it simple!
I think given example is trivial and that's why answer for specific example is 2nd method. But there are time's when first method is far more easy to implement. For example you create pages for website dynamically from admin panel, and in start you don't know all the values that will be used in every page. So you put general options like in 2nd method, and put something like page_data and use it to store serialized object. Now you should use serialized object for data that are not likely to change individually, as they are treated as single piece of data.
In your code you fetch serialized object, do unserialize and use them as normal. This way you can add page specific data that are not generalized for every page, but still the page's are the same.
I have a table that stores specific updates for all customers.
Some sample table:
record_id | customer_id | unit_id | time_stamp | data1 | data2 | data3 | data4 | more
When I created the application, I did not realize how much this table would grow -- currently I have over 10mil records within 1 month. I am facing issues, when php stops executing due to amount of time it takes. Some queries produce top-1 results, based on the time_stamp + customer_id + unit_id
How would you suggest handling this type of issues? For example, I can create new table for each customer, although I think it does not a good solution.
I am stuck with no good solution in mind.
If you're on the cloud (where you're charged for moving data between server and db), ignore.
Move all logic to the server
The fastest query is a SELECT WHEREing the PRIMARY. It won't matter how large your database is, it will come back just as fast with a table of 1 row (as long as your hardware isn't unbalanced).
I can't tell exactly what you're doing with your query, but first download all of the sorting and limiting data into PHP. Once you've got what you need, SELECT the data directly WHEREing on record_id (I assume that's your PRIMARY).
It looks like your on demand data is pretty computationally intensive and huge, so I recommend using a faster language. http://blog.famzah.net/2010/07/01/cpp-vs-python-vs-perl-vs-php-performance-benchmark/
Also, when you start sorting and limiting on the server rather than the db, you can start identifying shortcuts to speed it up even further.
This is what the server's for.
I suggest you use partitioning of your data following some criteria.
You can make horizontal or vertical partition of your data.
For example group your customer_id in 10 partitions, using his id module 10.
So, customer_id terminated in 0 goes to partition 0, with ended in 1 goes to partition 1
MySQL can make this for you easily.
What is the count of records within the tables? Often, with relational databases, it's not how much data you have (millions are nothing to relational databases), it's how you're retrieving it.
From the look of your select, in fact, you probably just need to optimize the statement itself and avoid the multiple subselects, which is probably the main cause of the slowdown. Try running an explain on that statement, or just get the ids and run the interior select individually on the ids of the records that you've actually found & retrieved in the first run.
Just the fact that you have those subselects within your overall statement means that you haven't optimized that far into the process anyway. For example, you could be running a nightly or hourly cron job that aggregates into a new table the sets like the one created by SELECT gps_unit.idgps_unit, and then you can run your selects against a previously generated table instead of creating blocks of data that are equivalent of a table on the fly.
If you find yourself unable to effectively optimize that select statement, you have "final" options like:
Categorize via some criteria and split into different tables.
Keep a deep archive, such that anything past the first year or so is migrated to a less used table and requires special retrieval.
Finally, if you have so much small data, you may be able to completely archive certain tables and keep them around in file form only and then truncate past a certain date. Often with web tracking data that isn't that important and is kinda spammy, I end up doing this after a few years, when the data is really not going to do anyone any good any more.
it is just a simple question but i want to solve it as best as possible.
There will be a table in html, which would be filled with data from a mysql query,
for example:
name | street | zip
What i want is to make this changeable, so that user can directly change the results -> click on save -> Update via mysql.
My plan was now, to display the data in input fields which are named like this:
name_id | street_id | zip_id
By clicking on save i would perform a while-loop to get all names, streets, and zips of the id and perform an mysql update afterwords. I know that this would be possible that way (i already used this 1 time).
But: is there a more easier and better solution for this problem?
BR
If you plan to use javascript this is the way to do it!
I was just working with this a few hours ago.
http://www.jeasyui.com/extension/edatagrid.php
Updating all elements could cause heavy load on your server, depending on the amount of data you have stored. A more performance-friendly solution would be to "save" the row keys of the data that you changed. If you do this, you would only update the rows that actually changed, and not all rows!
When I try and run this query on my database:
$mysqli->query("UPDATE
catalogsearch_fulltext
SET data_index = '".$unique_string."'
WHERE MATCH(data_index) AGAINST ('".$savestring."')
AND store_id = '1'");
It not only doesn't work, but it clears my ENTIRE catalogsearch_fulltext table. What am I doing wrong? I'm somewhat new to SQL so this is hopefully a quick error someone can spot.
I can provide more code if requested, but I don't think it is applicable.
Thanks!
Edit:
$unique_string = "325-FBA-BA-0006-x3|Badger|Enabled|Taxable Goods|Badger - SPF 30+ Anti-Bug Sunscreen, 2.9oz. - (Pack of 3)|47300|You asked, and Badger listened! After receiving many requests for a totally natural combination bug repellent and sunscreen, Badger took their wildly popular SPF 30+ Sunscreen formula and combined it with their Anti-Bug Balm formula to create easy-to-apply, safe, and natural protection for the whole family!|43.35|1"
$string = "the same as above but with duplicat entries in the | Pipe | sections."
It is also worth noting that this query is in a loop that goes through about 6 records.
If $unique_string and $savestring were both uninitialised, I expect you would set your index column to null - which would result in them not showing up in any selects that use that index column?
Perhaps you meant $uniquestring and $save_string? or perhaps these variables are out of scope?