Kind of strange question, I assume PHP is faster but maybe someone knows better.
Question is the following:
A table with approximately 10k rows, varchar(550) field. Field is indexed.
Select that retrieves data has a lot of OR-cases and does a LIKE scan for every condition.
Query is run from PHP script.
Would it be faster to lowercase search substring before running query in script?
e.g.
SELECT * FROM table WHERE field LIKE LOWER('%substring_1%') OR field LIKE LOWER ('%substring_2%') OR ... field LIKE LOWER ('%substring_100%')
Or something similar to:
//$substr_array contains all substrings
$condition = "";
foreach ($substr_array as $substring) {
$condition .= "field LIKE '%".mb_strtolower($substring, "UTF-8")."%' OR ";
}
$condition = substr($condition, 0, -4);
$query = "SELECT * FROM table WHERE $condition";
$result = query($query);
This is an opinion based question, but a question nonetheless. So here goes my answer:
SELECT *
FROM table
WHERE field LIKE LOWER('%substring_1%')
OR field LIKE LOWER ('%substring_2%')
OR ... field LIKE LOWER ('%substring_100%')
this scenario depends on how well you DB is structured. If you don't have a lot of records in your database then it should not matter giving the "outstanding" performances that we have nowadays...
However: "Another way for case-insensitive matching is to use a different “collation”. The default collations used by SQL Server and MySQL do not distinguish between upper and lower case letters—they are case-insensitive by default." (extract from use-the-index-luke)
You should read that article, it's very good.
Now down to the second part. The most expensive, as a performance cost, in the PHP code will be the foreach() statement. HERE you can find more info on how much time does a foreach() loop takes and a very good comparison with for() and while() loops. It's good to know them. So basically, what you do here is first take some time preparing the query, which costs time, and then actually running it, which costs time.
In conclusion, the real battle here will be: Which is faster? A foreach() or the LOWER() function in SQL.
Providing that I gave you some good materials to read in, and form your own conclusions, in my opinion, I think that the best way to go about this is to just leave the query and drop the foreach(), because queryes are the most cost-expensive and adding a foreach() will just slow you performance down. At least you cut something out of the picture. :)
I really hope that this helps you!
Keep on coding!
Ares.
Related
For example, i have a table "tbl_book" with 100 records or more with multiple column like book_name, book_publisher,book_author,book_rate in mysql "db_bookshop". Now i would like to fetch them all by one query without iterate 100 times instead of one or two time looping. Is it possible? Is there any tricky way to do that. Generally we do what
$result = mysql_query("SELECT desire_column_name FROM table_name WHERE clause");
while( $row = mysql_fetch_array($result) ) {
$row['book_name'];
$row['book_publisher'];
$row['book_author'];
..........
$row['book_rate'];
}
// Or we may can use mysqli_query(); -mysqli_fetch_row(), mysqli_fetch_array(), mysqli_fetch_assoc();
My question is, is there any idea or any tricky way that we can be
avoided 1oo times iterate for fetching 1oo records? It's may be wired
to someone but one of the most experience programmer told me that it's
possible. But unfortunately i was not able to learn it from him. I
feel sorry for him because he is not anymore. Advance thanks for your idea sharing.
You should not use mysql_query the mysql extension is deprecated:
This extension is deprecated as of PHP 5.5.0, and has been removed as of PHP 7.0.0.
-- https://secure.php.net/manual/en/intro.mysql.php
When you use PDO you can fetch all items without looping over query like this
$connection = new PDO('mysql:host=localhost;dbname=testdb', 'dbuser', 'dbpass');
$statement = $connection->query('SELECT ...');
$rows = $statement->fetchAll();
The short answer - NO, it's impossible to fetch more than one record from a database without a loop.
But the the question here is that you don't want it.
There is no point in "just fetching" the data - you're always going to do something with it. With each row. Obviously, a loop is a natural way to do something with each row. Therefore, there is no point in trying to avoid a loop.
Which renders your question rather meaningless.
Regarding performance. The truth is that you experience not a single performance problem related to fetching just 100 records from a database. Which renters your problem an imaginary one.
The only plausible question I can think off your post is your performance as a programmer, as lack of education makes you write a lot of unnecessary code. If you manage to ask a certain question regarding that matter, you'll be shown a way to avoid the useless repetitive typing.
Have you tried using mysql_fetch_assoc?
$result = mysql_query("SELECT desire_column_name FROM table_name WHERE clause");
while ($row = mysql_fetch_assoc($result)) {
// do stuff here like..
if (!empty($row['some_field'])){
echo $row["some_field"];
}
}
It is possible to read all 100 records without loop by hardcoding the main column values, but that would involve 100 x number of columns to be listed, and there could be limitation on the number of columns you can display in MySQL.
eg,
select
case when book_name='abc' then book_name end Name,
case when book_name='abc' then book_publisher end as Publisher,
case when book_name='abc' then book_author end as Author,
case when book_name='xyz' then book_name end Name,
case when book_name='xyz' then book_publisher end as Publisher,
case when book_name='xyz' then book_author end as Author,
...
...
from
db_bookshop;
It's not practical but if you have less rows to query you might find it useful.
The time taken to ask the MySQL server for something is far greater than one iteration through a client-side WHILE loop. So, to improve performance, the goal is to have the SELECT go to the server in one round trip. Different API calls do this or don't do this; read their details.
I have written a lot of UIs with MySQL under the covers. I think nothing of fetching a few dozen rows at once, and then build a <table> (or something) with the results. I rarely fetch more than 100, not because of performance, but because 100 is (usually) too much for the user to take in on a single web page.
Also, I think nothing of issuing several, maybe dozens, of queries in support of a single web page. The delay is insignificant, especially when compared to the user's time for reading, digesting, and moving to the next page. So, I try to give the user a digestible amount of info without having to click to another page to get more. There are tradeoffs.
When it is practical to have SQL do the 'digesting', do so. It is faster for MySQL do do a SUM() and return just the total, rather than return dozens of rows for the client to add up. This is mostly a 'bandwidth' issue. Either way, MySQL will fetch (internally) all the needed rows.
I have been working on laravel using eloquent query builders. I have a situation in which I am filtering records based on search queries. I have an integer field, for which I want to filter data in ranges. User can select any of available ranges for example;
0-15, 15-30 and 30 and above.
For this purpose I found Query-builders whereBetween() clause very helping. But It becomes difficult for me for last option when I want to select for 30 and above.
I would really appreciate if someone could help me with some trick make this query
->whereBetween('time_taken', [$lower-limit,$uper_limt])
working for all cases.
I don't want to write an additional line for this case, at which I can use simple where clause
->where('time_taken','>=',$uper_limt).
The practical solution here is to just choose the appropriate SQL condition (I'm using a ternary operator here to keep it more compact):
$query = App\Operation::query();
(empty($upper_limit)) ? $query->where('time_taken','>=', $lower_limit)
: $query->whereBetween('time_taken', [$lower_limit, $upper_limit]);
$results = $query->get();
Sure it would be nice to have just one line:
$results = App\Operation::whereBetween('time_taken', [$lower_limit, $upper_limit])->get();
But that's not possible in this case, not unless you want to extend the Laravel Query Builder and modify the way it handles empty parameters in the range passed as the value.
Writing clean and concise code is something we all strive to achieve, but one line solutions are not always possible. So my advice is to stop fixating (something I sometimes do myself) on things like this, because in some cases it's a lost cause that just ends up wasting time.
You can try any of these:
Method 1:
->whereBetween('time_taken', [$lower-limit,ModelName::max('time_taken')])
Method 2:
->whereBetween('time_taken', [$lower-limit,DB::table('table_name')->max('time_taken')])
Method 3:
$max = ModelName::max('time_taken');
//or
$max = DB::table('table_name')->max('time_taken');
//then
->whereBetween('time_taken', [$lower-limit,$max])
max() returns the highest value from your corespondent column.
I'm not sure that I have the terminology correct but basically I have a website where members can subscribe to topics that they like and their details go into a 'favorites' table. Then when there is an update made to that topic I want each member to be sent a notification.
What I have so far is:
$update_topic_sql = "UPDATE topics SET ...My Code Here...";
$update_topic_res = mysqli_query($con, $update_topic_sql)or die(mysqli_error());
$get_favs_sql = "SELECT member FROM favourites WHERE topic = '$topic'";
$get_favs_res = mysqli_query($con, $get_favs_sql);
//Here I Need to update the Members selected above and enter them into a notes table
$insert_note_sql = "INSERT INTO notes ....";
Does anyone know how this can be achieved?
Ok, so we've got our result set of users. Now, I'm going to assume from the nature of the question that you may be a bit of a newcomer to either PHP, SQL(MySQL in this case), web development, or all of the above.
Your question part 1:
I have no idea how to create an array
This is easier than what you may think, and if you've already tried this then I apologize, I don't want to insult your intelligence. :)
Getting an array from a mysqli query is just a matter of a function call and a loop. When you ran your select query and saved the return value to a variable, you stored a mysqli result set. The mysqli library supports both procedural and object oriented styles, so since you're using the procedural method, so will I.
You've got your result set
$get_favs_res = mysqli_query($con, $get_favs_sql);
Now we need an array! At this point we need to think about exactly what our array should be of, and what we need to do with the contents of the request. You've stated that you want to make an array out of the results of the SELECT query
For the purposes of example, I'm going to assume that the "member" field you've returned is an ID of some sort, and therefore a numeric type, probably of type integer. I also don't know what your tables look like, so I'll be making some assumptions there too.
Method 1
//perform the operations needed on a per row basis
while($row = mysqli_fetch_assoc($get_favs_res)){
echo $row['member'];
}
Method 2
//instead of having to do all operations inside the loop, just make one big array out of the result set
$memberArr = array();
while($row = mysqli_fetch_assoc($get_favs_res)){
$memberArr[] = $row;
}
So what did we do there? Let's start from the beginning to give you an idea of how the array is actually being generated. First, the conditional in the while loop. We're setting a variable as the loop condition? Yup! And why is that? Because when PHP (and a lot of other languages) sets that variable, the conditional will check against the value of the variable for true or false.
Ok, so how does it get set to false? Remember, any non boolean false, non null, non 0 (assuming no type checking) resolves to true when it's assigned to something (again, no type checking).
The function returns one row at a time in the format of an associative array (hence the _assoc suffix). The keys to that associative array are simply the names of the columns from the row. So, in your case, there will be one value in the row array with the name "member". Each time mysqli_fetch_assoc() is called with your result set, a pointer is pointed to the next result in the set (it's an ordered set) and the process repeats itself. You essentially get a new array each time the loop iterates, and the pointer goes to the next result too. Eventually, the pointer will hit the end of the result set, in which case the function will return a NULL. Once the conditional picks up on that NULL, it'll exit.
In the second example, we're doing the exact same thing as the first. Grabbing an associative array for each row, but we're doing something a little differently. We're constructing a two dimensional array, or nested array, of rows. In this way, we can create a numerically indexed array of associative arrays. What have we done? Stored all the rows in one big array! So doing things like
$memberArr[0]['member'];//will return the id of the first member returned
$memberArr[1]['member'];//will return the id of the second member returned
$lastIndex = sizeof($memberArr-1);
$memberArr[$lastIndex]['member'];//will return the id of the last member returned.
Nifty!!!
That's all it takes to make your array. If you choose either method and do a print_r($row) (method 1) or print_r($memberArr) (method 2) you'll see what I'm talking about.
You question part 2:
Here I Need to update the Members selected above and enter them into a notes table
This is where things can get kind of murky and crazy. If you followed method 1 above, you'd pretty much have to call
mysqli_query("INSERT INTO notes VALUES($row['member']);
for each iteration of the loop. That'll work, but if you've got 10000 members, that's 10000 inserts into your table, kinda crap if you ask me!
If you follow method two above, you have an advantage. You have a useful data structure (that two dim array) that you can then further process to get better performance and make fewer queries. However, even from that point you've got some challenges, even with our new processing friendly array.
The first thing you can do, and this is fine for a small set of users, is use a multi-insert. This just involves some simple string building (which in and of itself can pose some issues, so don't rely on this all the time) We're gonna build a SQL query string to insert everything using our results. A multi insert query in MySQL is just like a normal INSERT except for one different: INSERT INTO notes VALUES (1),(2),(x)
Basically, for each row you are inserted, you separate the value set, that set delineated by (), with a comma.
$query = 'INSERT INTO notes VALUES ';
//now we need to iterate over our array. You have your choice of loops, but since it's numerically indexed, just go with a simple for loop
$numMembers = sizeof($memberArr);
for($i = 0; $i < $numMembers; $i++){
if($i > 0){
$query .= ",({$membersArr[$i]['member']})";//all but the first row need a comma
}
else {
$query .= "({$membersArr[$i]['member']})";//first row does not need a comma
}
}
mysqli_query($query);//add all the rows.
Doesn't matter how many rows you need to add, this will add them. However, this is still going to be a costly way to do things, and if you think your sets are going to be huge, don't use it. You're going to end up with a huge string, TONS of string concats, and an enormous query to run.
However, given the above, you can do what you're trying to do.
NOTE: These are grossly simplified ways of doing things, but I do it this way because I want you to get the fundamentals down before trying something that's going to be way more advanced. Someone is probably going to comment on this answer without reading this note telling me I don't know what I'm doing for going about this so dumbly. Remember, these are just the basics, and in no way reflect industrial strength techniques.
If you're curious about other ways of generating arrays from a mysqli result set:
The one I used above
An even easier way to make your big array but I wanted to show you the basic way of doing things before giving you the shortcuts. This is also one of those functions you shouldn't use much anyway.
Single row as associative(as bove), numeric, or both.
Some folks recommend using loadfiles for SQL as they are faster than inserts (meaning you would dump out your data to a file, and use a load query instead of running inserts)
Another method you can use with MySQL is as mentioned above by using INSERT ... SELECT
But that's a bit more of an advanced topic, since it's not the kind of query you'd see someone making a lot. Feel free to read the docs and give it a try!
I hope this at least begins to solve your problem. If I didn't cover something, something didn't make sense, or I didn't your question fully, please drop me a line and I'll do whatever I can to fix it for you.
Quite simply, is such a query/statement inefficient or bad?
<?
$strSql="SELECT * FROM clients, projects
WHERE clients.clientID = $intClientId
AND projects.clientID=$intClientId LIMIT 1";
$objResult=mysql_query($strSql);
if(mysql_num_rows($objResult)==0) {
echo("No data"); }
while ($arrRow=mysql_fetch_array($objResult))
{
?>
<h1>Sub Project(s) for: <span><?=$arrRow[clientName]?></span></h1>
<?
} ?>
In general, you should avoid using SELECT * and select only the fields you need unless absolutely necessary. Whether or not this is efficient depends on how your tables are indexed. I assume in this case that that clientID is the primary key of the clients table. If you have an index on clientID in the projects table, this query should be quite fast.
There are several things that are off here:
As Michael Mior mentions, you should avoid SELECT *. This can be an efficiency issue. It can also break applications in some cases if your application makes assumptions about the columns that are in the table and then the table changes in the database.
You have a LIMIT 1 in your query, but then you loop over the results. This doesn't make sense because LIMIT 1 means you'll only get one row of results, regardless of how many rows matched your query.
You are not escaping your inputs. This may be OK in this case if, in earlier code, you have already verified that those variables definitely contain integer values. I generally just use prepared statements and avoid this problem altogether.
And i also want to add one little advice: Try not to use php shorttags(Use <?php echo instead of <?=) from now. Since PHP6 it will not be supported and it might create code errors and other difficulties for you in the future.
When writing, or should I say developing new features, one should always debug and run a performance profile on query to ensure that you'r using proper index(s). Always use EXPLAIN (or EXPLAIN EXTENDED if your into specifics) on a query to determine its performance.
EXPLAIN SELECT * FROM clients, projects WHERE clients.clientID = 1 AND projects.clientID = 1
I noted also that your doing a while() loop, which is unnecessary if your are only fetching a single row.
I prefer writing projects.clientID=clients.clientID so to be more easy to see how the tables are joined.
The other thing I will avoid (mostly for clarity) is the while loop. Since you expect one record, no need to loop over the result set.
And finally, it is not a good idea to use the short tags.
I need to pull several rows from a table and process them in two ways:
aggregated on a key
row-by-row, sorted by the same key
The table looks roughly like this:
table (
key,
string_data,
numeric_data
)
So I'm looking at two approaches to the function I'm writing.
The first would pull the aggregate data with one query, and then query again inside a loop for each set of row-by-row data (the following is PHP-like pseudocode):
$rows = query(
"SELECT key,SUM(numeric_data)
FROM table
GROUP BY key"
);
foreach ($rows as $row) {
<process aggregate data in $row>
$key = $row['key'];
$row_by_row_data = handle_individual_rows($key);
}
function handle_individual_rows($key)
{
$rows = query(
"SELECT string_data
FROM table WHERE key=?",
$key
);
<process $rows one row at a time>
return $processed_data;
}
Or, I could do one big query and let the code do all the work:
$rows = query(
"SELECT key, string_data, numeric_data
FROM table"
);
foreach ($rows as $row) {
<process rows individually and calculate aggregates as I go>
}
Performance is not a practical concern in this application; I'm just looking to write sensible and maintainable code.
I like the first option because it's more modular -- and I like the second option because it seems structurally simple. Is one option better than the other or is it really just a matter of style?
One SQL query, for sure.
This will
Save you lots of roundtrips to database
Allow to use more efficient GROUP BY methods
Since your aggregates may be performed equally well by the database, it will also be better for mainainability: you have all your resultset logic in one place.
Here is an example of a query that returns every row and calculates a SUM:
SELECT string_data, numeric_data, SUM(numeric_data) OVER (PARTITION BY key)
FROM table
Note that this will most probably use parallel access to calculate SUM's for different key's, which is hardly implementable in PHP.
Same query in MySQL:
SELECT key, string_data, numeric_data,
(
SELECT SUM(numeric_data)
FROM table ti
WHERE ti.key = to.key
) AS key_sum
FROM table to
If performance isn't a concern, I'd go with the second. Seems the tiniest bit friendlier.
If performance were a concern, my answer would be "don't think, profile". :)
The second answer is by far more clear, sensible and maintainable. You're saying the same thing with less code, which is usually better.
And I know you said performance is not a concern, but why fetch data more than you have to?
I can't be certain from the example here, but I'd like to know if there's a chance to do the aggregation and other processing right in the SQL query itself. In this case, you'd have to evaluate "more maintainable" with respect to your relative comfort level expressing that processing in SQL code vs. PHP code.
Is there something about the additional processing you need to do on each row that would prevent you from expressing everything in the SQL query itself?
I don't think you'll find many situations at all where doing a query-per-iteration of a loop is the better choice. In fact, I'd say it's probably a good rule of thumb to never do that.
In other words, the fewer round trips to the database, the better.
Depending on your data and actual tables, you might be able to let SQL do the aggregation work and select all the rows you need with one query.
one sql query is probably a better idea.
It avoids you having to re-write relational operations
I think somehow you've answered your own question, because you say you have two different processings : one aggregation and one row by row.
if you want to keep everything readable and maintainable, mixing both in a single query doesn't sound right, the query will answer two different needs so it won't be very readable
even if perf is not an issue, it's faster to do the aggregation on the DB server instead of doing it in code
with only one query, the code that will handle the result will mix two processings, handling rows and computing aggregations in the same time, so in time this code will tend to get confusing and buggy
the same code might evolve over time, for instance the row-by-row can get complex and could create bugs in the aggregation part or the other way around
if in the future you'll need to split these two treatments, it will be harder to disentangle the code that at that moment, somebody else has written ages ago...
Performance considerations aside, in terms of maintainability and readability I'd recommend to use two queries.
But keep in mind that the performance factor might not be an issue at the moment, but it can be in time once the db volume grows or whatever, it's never a negligible factor on long term ...
Even if perf is not an issue, your mind is. When a musician practices every movement is intended to improve the musician's skill. As a developer, you should develop every procedure to improve your skill. iterative loops though data is sloppy and ugly. SQL queries are elegant. Do you want to develop more elegant code or more sloppy code?