MySQL query order by "most completed fields" - php

Question

I have a table witch has 45 columns but only a few of these are yet completed. This table is continuously updated and added etc. In my auto-complete function i want to select these records ordered by the most completed fields(i hope you understand)?
One of the solutions is to create another filed (the "rank" field) and create a php function that selects * the records and gives a rank for each record.
... but i was wondering if there is a more simple way of doing this only whit a single ORDER BY?

Answer 1

MySQL has no function to count the number of non-NULL fields on a row, as far as I know.
So the only way I can think of is to use an explicit condition:
SELECT * FROM mytable
ORDER BY (IF( column1 IS NULL, 0, 1)
+IF( column2 IS NULL, 0, 1)
...
+IF( column45 IS NULL, 0, 1)) DESC;
...it is ugly as sin, but should do the trick.
You could also devise a TRIGGER to increment an extra column "fields_filled". The trigger costs you on UPDATE, the 45 IFs hurt you on SELECT; you'll have to model what is more convenient.
Note that indexing all fields to speed up SELECT will cost you when updating (and 45 different indexes probably cost as much as a table scan on select, not to say that the indexed field is a VARCHAR). Run some tests, but I believe that the 45-IF solution is likely to be the best overall.
UPDATE:
If you can rework your table structure to normalize it somewhat, you could put the fields in a my_values table. Then you would have a "header table" (maybe with only a unique ID) and a "data table". Empty fields would not exist at all, and then you could sort by how many filled fields are there by using a RIGHT JOIN, counting the filled fields with COUNT(). This would also greatly speed up UPDATE operations, and would allow you to efficiently employ indexes.
EXAMPLE (from table setup to two normalized tables setup):
Let us say we have a set of Customer records. We will have a short subset of "mandatory" data such as ID, username, password, email, etc.; then we will have a maybe much larger subset of "optional" data such as nickname, avatar, date of birth, and so on. As a first step let us assume that all these data are varchar (this, at first sight, looks like a limitation when compared to the single table solution where each column may have its own datatype).
So we have a table like,
ID username ....
1 jdoe etc.
2 jqaverage etc.
3 jkilroy etc.
Then we have the optional-data table. Here John Doe has filled all fields, Joe Q. Average only two, and Kilroy none (even if he was here).
userid var val
1 name John
1 born Stratford-upon-Avon
1 when 11-07-1974
2 name Joe Quentin
2 when 09-04-1962
In order to reproduce the "single table" output in MySQL we have to create a quite complex VIEW with lots of LEFT JOINs. This view will nonetheless be very fast if we have an index based on (userid, var) (even better if we use a numeric constant or a SET instead of a varchar for the datatype of var:
CREATE OR REPLACE VIEW usertable AS SELECT users.*,
names.val AS name // (1)
FROM users
LEFT JOIN userdata AS names ON ( users.id = names.id AND names.var = 'name') // (2)
;
Each field in our logical model, e.g., "name", will be contained in a tuple ( id, 'name', value ) in the optional data table.
And it will yield a line of the form <FIELDNAME>s.val AS <FIELDNAME> in the section (1) of the above query, referring to a line of the form LEFT JOIN userdata AS <FIELDNAME>s ON ( users.id = <FIELDNAME>s.id AND <FIELDNAME>s.var = '<FIELDNAME>') in section (2). So we can construct the query dynamically by concatenating the first textline of the above query with a dynamic Section 1, the text 'FROM users ' and a dynamically-built Section 2.
Once we do this, SELECTs on the view are exactly identical to before -- but now they fetch data from two normalized tables via JOINs.
EXPLAIN SELECT * FROM usertable;
will tell us that adding columns to this setup does not slow down appreciably operations, i.e., this solution scales reasonably well.
INSERTs will have to be modified (we only insert mandatory data, and only in the first table) and UPDATEs as well: we either UPDATE the mandatory data table, or a single row of the optional data table. But if the target row isn't there, then it must be INSERTed.
So we have to replace
UPDATE usertable SET name = 'John Doe', born = 'New York' WHERE id = 1;
with an 'upsert', in this case
INSERT INTO userdata VALUES
( 1, 'name', 'John Doe' ),
( 1, 'born', 'New York' )
ON DUPLICATE KEY UPDATE val = VALUES(val);
(We need a UNIQUE INDEX on userdata(id, var) for ON DUPLICATE KEY to work).
Depending on row size and disk issues, this change might yield an appreciable performance gain.
Note that if this modification is not performed, the existing queries will not yield errors - they will silently fail.
Here for example we modify the names of two users; one does have a name on record, the other has NULL. The first is modified, the second is not.
mysql> SELECT * FROM usertable;
+------+-----------+-------------+------+------+
| id | username | name | born | age |
+------+-----------+-------------+------+------+
| 1 | jdoe | John Doe | NULL | NULL |
| 2 | jqaverage | NULL | NULL | NULL |
| 3 | jtkilroy | NULL | NULL | NULL |
+------+-----------+-------------+------+------+
3 rows in set (0.00 sec)
mysql> UPDATE usertable SET name = 'John Doe II' WHERE username = 'jdoe';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> UPDATE usertable SET name = 'James T. Kilroy' WHERE username = 'jtkilroy';
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0 Changed: 0 Warnings: 0
mysql> select * from usertable;
+------+-----------+-------------+------+------+
| id | username | name | born | age |
+------+-----------+-------------+------+------+
| 1 | jdoe | John Doe II | NULL | NULL |
| 2 | jqaverage | NULL | NULL | NULL |
| 3 | jtkilroy | NULL | NULL | NULL |
+------+-----------+-------------+------+------+
3 rows in set (0.00 sec)
To know the rank of each row, for those users that do have a rank, we simply retrieve the count of userdata rows per id:
SELECT id, COUNT(*) AS rank FROM userdata GROUP BY id
Now to extract rows in "filled status" order, we do:
SELECT usertable.* FROM usertable
LEFT JOIN ( SELECT id, COUNT(*) AS rank FROM userdata GROUP BY id ) AS ranking
ON (usertable.id = ranking.id)
ORDER BY rank DESC, id;
The LEFT JOIN ensures that rankless individuals get retrieved too, and the additional ordering by id ensures that people with identical rank always come out in the same order.