Most effective way of data collection? - php

Question

Let's first get to an important note about my situation:
I have 1 table in my MySQL database with approx 10 thousand entries
Currently, when collecting information from table #1. I collect a total of 20 - 24 rows per page.
Example being:
Q1 : SELECT * FROM table WHERE cat = 1 LIMIT 0,25
R1: id: 1, name: something, info: 12
The PHP file that does these queries, is called by the jquery ajax function, and creates an XML file that that jquery function reads and shows to the user.
My question here is. How do i improve the speed & stability of this process. I can have up to 10 thousand visitors picking up information at the same time, which makes my server go extremely sluggish and in some cases even crash.
I'm pretty much out of idea's, so i'm asking for help here. Here's an actual presentation of my current data collection (:
public function collectItems($type, $genre, $page = 0, $search = 0)
{
// Call Core (Necessary for Database Interaction
global $plusTG;
// If Search
if($search)
{
$searchString = ' AND (name LIKE "%'.$search.'%")';
}
else
{
$searchString = '';
}
// Validate Query
$search = $plusTG->validateQuery($search);
$type = $plusTG->validateQuery($type);
$genre = $plusTG->validateQuery($genre);
// Check Numeric
if((!is_numeric($genre)))
{
return false;
}
else
{
if(!is_numeric($type))
{
if($type != 0)
{
$typeSelect = '';
$split = explode(',',$type);
foreach($split as $oneType)
{
if($typeSelect == '')
{
$typeSelect .= 'type = '.$oneType.' ';
}
else
{
$typeSelect .= 'OR type = '.$oneType.' ';
}
}
}
}
else
{
$typeSelect = 'type = ' . $type . ' ';
}
//echo $typeSelect;
$limit = ($page - 1) * 20;
if(($type != 0) && ($genre != 0))
{
$items = $plusTG->db->query('SELECT * FROM dream_items WHERE active = 1 AND genre = '.$genre.' AND ('.$typeSelect.')'.$searchString.' ORDER BY name LIMIT '.$limit.',20');
$total = $plusTG->db->query('SELECT COUNT(*) as items FROM dream_items WHERE active = 1 AND genre = '.$genre.' AND ('.$typeSelect.')'.$searchString);
}
elseif(($type == 0) && ($genre != 0))
{
$items = $plusTG->db->query('SELECT * FROM dream_items WHERE active = 1 AND genre = '.$genre.$searchString.' ORDER BY name LIMIT '.$limit.',20');
$total = $plusTG->db->query('SELECT COUNT(*) as items FROM dream_items WHERE active = 1 AND genre = '.$genre.$searchString);
}
elseif(($type != 0) && ($genre == 0))
{
$items = $plusTG->db->query('SELECT * FROM dream_items WHERE active = 1 AND ('.$typeSelect.')'.$searchString.'ORDER BY name LIMIT '.$limit.',20');
$total = $plusTG->db->query('SELECT COUNT(*) as items FROM dream_items WHERE active = 1 AND ('.$typeSelect.')'.$searchString);
}
elseif(($type == 0) && ($genre == 0))
{
$items = $plusTG->db->query('SELECT * FROM dream_items WHERE active = 1'.$searchString.' ORDER BY name LIMIT '.$limit.',20');
$total = $plusTG->db->query('SELECT COUNT(*) as items FROM dream_items WHERE active = 1'.$searchString);
}
$this->buildInfo($items->num_rows, $total->fetch_assoc());
while($singleItem = $items->fetch_assoc())
{
$this->addItem($singleItem);
}
}
return true;
}
The build info call & add item call are adding the items to the DOMXML.
This is my javascript (domain and filename filtered):
function itemRequest(type,genre,page, search)
{
if(ajaxReady != 0)
{
ajaxReady = 0;
$('#item_container').text('');
var searchUrl = '';
var searchLink;
var ajaxURL;
if(search != 0)
{
searchUrl = '&search=' + search;
searchLink = search;
ajaxURL = "/////file.php";
}
else
{
searchLink = 0;
ajaxURL = "////file.php";
}
$.ajax({
type: "GET",
url: ajaxURL,
data: "spec=1&type="+type+"&genre="+genre+"&page="+page+searchUrl,
success: function(itemListing){
$(itemListing).find('info').each(function()
{
var total = $(this).find('total').text();
updatePaging(total, page, type, genre, searchLink);
});
var items = $(itemListing).find('items');
$(items).find('item').each(function()
{
var itemId = $(this).find('id').text();
var itemType = $(this).find('type').text();
var itemGenre = $(this).find('genre').text();
var itemTmId = $(this).find('tm').text();
var itemName = $(this).find('name').text();
buildItem(itemId, itemType, itemGenre, itemTmId, itemName);
});
$('.item_one img[title]').tooltip();
},
complete: function(){
ajaxReady = 1;
}
});
}
Build item calls this:
function buildItem(itemId, itemType, itemGenre, itemTmId, itemName)
{
// Pick up Misc. Data
var typeName = nameOfType(itemType);
// Create Core Object
var anItem = $('<div/>', {
'class':'item_one'
});
// Create Item Image
$('<img/>', {
'src':'///'+typeName+'_'+itemTmId+'_abc.png',
'alt':itemName,
'title':itemName,
click:function(){
eval(typeName + 'Type = ' + itemTmId);
$('.equipped_item[name='+typeName+']').attr('src','//'+typeName+'_'+itemTmId+'_abc.png');
$('.equipped_item[name='+typeName+']').attr('alt',itemName);
$('.equipped_item[name='+typeName+']').attr('title',itemName);
$('.equipped_item[title]').tooltip();
recentEquipped(typeName, itemTmId, itemName);
updateSelfy();
}
}).appendTo(anItem);
// Favs
var arrayHack = false;
$(favEquips).each(function(){
if(arrayHack == false)
{
if(in_array(itemTmId, this))
{
arrayHack = true;
}
}
});
var itemFaved = '';
if(arrayHack == true)
{
itemFaved = 'activated';
}
$('<div/>',{
'class':'fav',
'id':itemFaved,
click:function(){
if($(this).attr('id') != 'activated')
{
$(this).attr('id','activated');
}
else
{
$(this).removeAttr('id');
}
itemFav(itemTmId, typeName, itemName);
}
}).appendTo(anItem);
$(anItem).appendTo('#item_container');
}
If anyone could help me improve this code, it'd be very much appreciated.

Answer 1

add an index to your table for cat column
figure out what the bottleneck is, if it is your XML then try json,
if it is your network, try enabling gzip compression

Answer 2

I agree with Zepplock, it is important to find out where the bottleneck is - if not, you're only guessing. Zepplock's list is good but I would also add caching:
Find out where the bottleneck is.
Use indexes in your db table.
Cache your query results
Find the Bottleneck.
There are number opinions and ways to do this... Basically when your site is under load, get the time it takes to complete each step in the process: The DB queries, the server-side processes, the client side processes.
Use Indexes.
If your DB is slow chances are you can get a lot of improvement by optimizing your queries. A table index may be in order... Use 'EXPLAIN' to help identify where indexes should be placed to optimize your queries:
EXPLAIN SELECT * FROM dream_items WHERE active = 1 AND (name LIKE "%foo%") ORDER BY name LIMIT 0,20;
(I bet an index on active and name would do the trick)
ALTER TABLE `dream_items` ADD INDEX `active_name` (`active` , `name`);
Also try to avoid using the wildcard '*'. Instead only ask for the columns you need. Something like:
SELECT `id`, `type`, `genre`, `tm`, `name` FROM `dream_items` WHERE...
Cache Your Results.
If the records in the DB have not changed then there is no reason to try an re-query the results. Use some sort of caching to reduce the load on your DB (memcached, flat file, etc..). Depending on the database class / utilities you're using it may already be capable of caching results.