I am trying to create promo codes in large batches (with php/mysql).
Currently my code looks something like this:
$CurrentCodesInMyDB = "asdfasdf,asdfsdfx"; // this is a giant comma delimited string of the current promo codes in the database.
$PromoCodes = "";
for($i=0;$i<=30000;$i++)
{
$NewCode = GetNewCode($PromoCodes, $CurrentCodesInMyDB );
$PromoCodes .= $NewCode . ","; //this string gets used to allow them to download a txt file
//insert $newcode into database here
}
function GetNewCode($CurrentList, $ExistingList)
{
$NewPromo = GetRandomString();
if(strpos($CurrentList, $NewPromo) === false && strpos($ExistingList, $NewPromo) === false)
{
return $NewPromo;
}
else
{
return GetNewCode($CurrentList, $ExistingList);
}
}
function GetRandomString()
{
return "xc34cv87"; //some random 8 character alphanumeric string
}
When I do batches in 10k, it seems to be ok. But the client would like to be able to generate 30k at a time. When I bump the loop up to 30k, I've been having issues. Are there any obvious performance tweaks that I could make or maybe a different way I could do this?
You probably don't need to have all 30,000 codes loaded into memory in a single giant string. Just create a table in your database, add a code unique field (either primary key or unique index) and insert random codes until you have 30,000 successful insertions.
What kind of issues specifically?
My advice is: don't store the codes in a a CSV format, instead create a new indexed column and store each code on its own row - also, use prepared queries.
Doing 60,000 strpos() on a ~250 KB string might not be the best idea ever...
If you don't want to do inserts inside the loop.(they are also expensive) use an array and the method in_array to check for the string. Look in the comments for the in_array function there is someone saying that you can achieve better performance using array_flip and then checking for the array key
http://www.php.net/manual/en/function.in-array.php#96198
Related
Due to a weird set of circumstances, I need to determine if a value exists in a known set, then take an action. Consider:
An included file will look like this:
// Start generated code
$set = array();
$set[] = 'foo';
$set[] = 'bar';
// End generated code
Then another file will look like this:
require('that_last_file.php');
if(in_array($value, $set)) {
// Do thing
}
As noted, the creation of the array will be from generated code -- a process will create a PHP file which will be included above the if statement with require.
How concerned should I be about the size of this mess -- both in bytes, and array values? It could easily get to 5,000 values. How concerned should I be with the overhead of a 5,000-value array? Is there a more efficient way to search for the value, other than using in_array on an array? How painful is including a 5,000-line file via require?
I know there are ultimately better ways of doing this, but my limitations are that the set creation and logic has to be in an included PHP file. There are odd technical restrictions that prevent other options (i.e. -- a database lookup).
A faster way would be:
if (array_flip($set)[$value] !== null) {
// Do thing
}
A 5000 value array really isn't that bad though if it's just strings
I am trying to learn PHP while I write a basic application. I want a process whereby old words get put into an array $oldWords = array(); so all $words, that have been used get inserted using array_push(oldWords, $words).
Every time the code is executed, I want a process that finds a new word from $wordList = array(...). However, I don't want to select any words that have already been used or are in $oldWords.
Right now I'm thinking about how I would go about this. I've been considering finding a new word via $wordChooser = rand (1, $totalWords); I've been thinking of using an if/else statement, but the problem is if array_search($word, $doneWords) finds a word, then I would need to renew the word and check it again.
This process seems extremely inefficient, and I'm considering a loop function but, which one, and what would be a good way to solve the issue?
Thanks
I'm a bit confused, PHP dies at the end of the execution of the script. However you are generating this array, could you also not at the same time generate what words haven't been used from word list? (The array_diff from all words to used words).
Or else, if there's another reason I'm missing, why can't you just use a loop and quickly find the first word in $wordList that's not in $oldWord in O(n)?
function generate_new_word() {
foreach ($wordList as $word) {
if (in_array($word, $oldWords)) {
return $word; //Word hasn't been used
}
}
return null; //All words have been used
}
Or, just do an array difference (less efficient though, since best case is it has to go through the entire array, while for the above it only has to go to the first word)
EDIT: For random
$newWordArray = array_diff($allWords, $oldWords); //List of all words in allWords that are not in oldWords
$randomNewWord = array_rand($newWordArray, 1);//Will get a new word, if there are any
Or unless you're interested in making your own datatype, the best case for this could possibly be in O(log(n))
I need help to find workaround for getting over memory_limit. My limit is 128MB, from database I'm getting something about 80k rows, script stops at 66k. Thanks for help.
Code:
$posibilities = [];
foreach ($result as $item) {
$domainWord = str_replace("." . $item->tld, "", $item->address);
for ($i = 0; $i + 2 < strlen($domainWord); $i++) {
$tri = $domainWord[$i] . $domainWord[$i + 1] . $domainWord[$i + 2];
if (array_key_exists($tri, $possibilities)) {
$possibilities[$tri] += 1;
} else {
$possibilities[$tri] = 1;
}
}
}
Your bottleneck, given your algorithm, is most possibly not the database query, but the $possibilities array you're building.
If I read your code correctly, you get a list of domain names from the database. From each of the domain names you strip off the top-level-domain at the end first.
Then you walk character-by-character from left to right of the resulting string and collect triplets of the characters from that string, like this:
example.com => ['exa', 'xam', 'amp', 'mpl', 'ple']
You store those triplets in the keys of the array, which is nice idea, and you also count them, which doesn't have any effect on the memory consumption. However, my guess is that the sheer number of possible triplets, which is for 26 letters and 10 digits is 36^3 = 46656 possibilities each taking 3 bytes just for key inside array, don't know how many boilerplate code around it, take quite a lot from your memory limit.
Probably someone will tell you how PHP uses memory with its database cursors, I don't know it, but you can do one trick to profile your memory consumption.
Put the calls to memory-get-usage:
before and after each iteration, so you'll know how many memory was wasted on each cursor advancement,
before and after each addition to $possibilities.
And just print them right away. So you'll be able to run your code and see in real time what and how seriously uses your memory.
Also, try to unset the $item after each iteration. It may actually help.
Knowledge of specific database access library you are using to obtain $result iterator will help immensely.
Given the tiny (pretty useless) code snippet you've provided I want to provide you with a MySQL answer, but I'm not certain you're using MySQL?
But
- Optimise your table.
Use EXPLAIN to optimise your query. Rewrite your query to put as much of the logic in the query rather than in the PHP code.
edit: if you're using MySQL then prepend EXPLAIN before your SELECT keyword and the result will show you an explanation of actually how the query you give MySQL turns into results.
Do not use PHP strlen function as this is memory inefficient - instead you can compare by treating a string as a set of array values, thus:
for ($i = 0; !empty($domainWord[$i+2]); $i++) {
in your MySQL (if that's what you're using) then add a LIMIT clause that will break the query into 3 or 4 chunks, say of 25k rows per chunk, which will fit comfortably into your maximum operating capacity of 66k rows. Burki had this good idea.
At the end of each chunk clean all the strings and restart, set into a loop
$z = 0;
while ($z < 4){
///do grab of data from database. Preserve only your output
$z++;
}
But probably more important than any of these is provide enough details in your question!!
- What is the data you want to get?
- What are you storing your data in?
- What are the criteria for finding the data?
These answers will help people far more knowledgable than me to show you how to properly optimise your database.
Let says that I stored dataID in json file with 1,000,000 records.
My zresults.json = {"dataID":["1","2","3", ... "1000000"]}z
I want to find ID "100000" in the array.
$file = file_get_contents('results.json');
$data = json_decode($file,true);
if(in_array('100000', $data['dataID']))
{
echo "found";
} else {
echo "not found";
}
It took about 0.6 sec. for the result.
Is there a faster way for searching in json array like this?
Please give me an example!
Thank you in advance.
Update:
Although sql would much faster but considered 1,000,000 record in one table the more record the more space! At least, static file reduced server load and less space.
It depends on how designed your system. Use it the right place and the right time!
Sure!
$stm = $pdo->prepare("SELECT 1 FROM data WHERE id = ?");
$stm->execute(array(100000));
if ($stm->fetchColumn())
{ echo "found"; } else { echo "not found"; }
you will need to import your array into database first.
Depending on the structure of the data in the results.json file you may be able to do a simple string search for example
$file = file_get_contents('results.json');
if(strpos($file, '"100000"') !== false)
{
echo 'found';
}
else
{
echo 'not found';
}
After benchmarking your method I got around 0.78 seconds (on my slow local system) however with this method I achieved around 0.03 seconds.
Like I say, it depends on your data structure but if it does permit you to use this method you'll see significant speed benefits.
Maybe it's possible to try to predict the outcome, you can then use in_array to search the value in a much smaller json.
Otherwise you could try alternative search algorithms, those can be complicated.
A document based database like mongo DB is what you should try instead of working with plain json files - if you are bound to the json format.
Note that mongoDB can keep the json object in memory where plain PHP solutions had to parse the file again and again.
I see three performance boosts:
less disk IO
less parsing
index based searches
Why don't you store the id as keys and then do:
if(isset($data['dataID']['100000'])){
// do something
}
Because checking if a key exists is alot faster than looping through the array. You can check out this link for further information:
List of Big-O for PHP functions
I have any array
$num_list = array(42=>'0',44=>'0',46=>'0',48=>'0',50=>'0',52=>'0',54=>'0',56=>'0',58=>'0',60=>'0');
and I want to change specific values as I go through a loop
while(list($pq, $oin) = mysql_fetch_row($result2)) {
$num_list[$oin] = $pq;
}
So I want to change like 58 to 403 rather then 0.
However I always end up getting just the last change and non of the earlier ones. So it always ends up being something like
0,0,0,0,0,0,0,0,0,403
rather then
14,19,0,24,603,249,0,0,0,403
How can I do this so it doesn't overwrite it?
Thanks
Well, you explicititly coded that each entry should be replaced with the values from the database (even with "0").
You could replace the values on non-zero-values only:
while(list($pq, $oin) = mysql_fetch_row($result2)) {
if ($pq !== "0") $num_list[$oin] = $pq;
}
I don't get you more clear, i thought your asking this only. Check this
while(list($pq, $oin) = mysql_fetch_row($result2)) {
if($oin==58) {
$num_list[$oin] = $pq;
}
}
In my simulated tests (although You are very scarce with information), Your code works well and produces the result that You want. Check the second query parameter, that You put into array - namely $pg, thats what You should get there 0,0,0,0,0...403 OR Other thing might be that Your $oin numbers are not present in $num_list keys.
I tested Your code with mysqli driver though, but resource extraction fetch_row is the same.
Bear in mind one more thing - if Your query record number is bigger than $numlist array, and $oin numbers are not unique, Your $numlist may be easily overwritten by the folowing data, also $numlist may get a lot more additional unwanted elements.
Always try to provide the wider context of Your problem, there could be many ways to solve that and help would arrive sooner.