I have this code on my website, this function is called on every webpage, but it's slow (I did a lot of research and without this function the TTFB is about 100ms, but with this it can be even 2 seconds).
The function replaces every text in [] to a link, if a match is found in the card database. E.g.: [Inner Fire] >> and the output on the website will be:
<a href/card/id/name" class="quality1">Inner Fire</a>
It's working really great, but there are 3000 cards in the database and this is slow. Anyone can come up with a better solution to speed the process up?
Thank you in advance.
Some clarifications before the code:
sql_query:
function sql_query($conn, $query)
{
return mysqli_query($conn, $query);
}
Similar function with sql_fetch.
char_convert: converts utf-8 characters to HTML entity (decimal)
function coloredcard($text)
{
global $conn;
$query = "SELECT id, quality, name, collectible FROM cards";
$result = sql_query($conn, $query);
while ($card = sql_fetch($result))
{
$name_replace = strtolower(str_replace(str_split("\' "), "-", $card['name']));
if ($card['collectible']!=0) //if collectible, replace [card_names]
{
$from = '['.char_convert($card['name']).']';
$to = ''.$card['name'].'';
$text = str_ireplace($from, $to, $text);
}
elseif ($card['collectible']==0) //if not collectible replace (noncollectible card names)
{
$from = '('.char_convert($card['name']).')';
$to = ''.$card['name'].'';
$text = str_ireplace($from, $to, $text);
}
}
return $text;
}
Please let me know if you need further information.
The best way to accelerate this code will be to limit the number of cards that need to be fetched from the database. I'm not going to write the code for you, but here's an outline of how that could work:
Extract all the card names which are [linked] in the page, e.g. using preg_match_all().
Perform a single SQL query to load all of those cards, using WHERE name IN ('name1', 'name2', 'name3', …).
Loop through the result of that query and perform replacements on the HTML where appropriate.
Just my 2 cents:
You are not going to display 3000+ cards at the same time do you? So why not implementing an infinite loader which requests only a bunch of them (10 or so) and then asks for more as the user scrolls down?
$query = "SELECT id, quality, name, collectible FROM cards LIMIT ".$offset.",10";
Solution no.2:
have another table in which you store which cards are needed on which page, something like:
cardpage(cardid, pageid)
and have a JOIN query between card and cardpage tables
You can use MySQL's own string functions to do the replacement stuff while fetching data, much faster than iterating in PHP:
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html
You're making 3000 calls to str_replace(). You can accomplish the same result in one. See the docs for str_replace(), notably that the first and second parameters can be arrays:
$search = ['things', 'to', 'search', 'for', ... ];
$replace = ['things', 'to', 'replace', 'with', ... ];
$output = str_replace($search, $replace, $input);
Also, cache the output so that you only have to perform the replacement once.
Related
I want to separate the area code from a phone number string by using a area code mysql database.
For example the string is 0349152023.
The endresult should be 03491 52023.
To get the endresult, i want to split the string and search every digit in database.
For example 0 and then 3 and then 4 and then take the last found result.
The code i have at the moment is only to prepare the phone number string for futher actions:
$phone1 = preg_replace('/[oO]/', '0', $phone-string);
$phone2 = preg_replace("/[^0-9]/", "", $phone1);
Then i use str_split to cut the string in pieces:
$searchArray = str_split($phone2);
Thanks for your help.
You may build an array containing all the area codes.
Then you may write something like this:
foreach ($area_codes as $code) {
if (substr($phone, 0, strlen($code)) == $code) {
$phone_string = substr($phone, 0, strlen($code))." ".substr($phone, strlen($code));
}
}
You can obviously add a controller in order to verify if the area code was found or not.
step 1: select all area codes from db and put them into an array $areaCodes
step 2: iterate over $areaCodes as $code and check if the phonenumber starts with $code. if it does, create a string that has a whitespace between the code and the rest of the number
$phonenumber = '0349152023';
$preparedPhonenumber = '';
foreach($areaCodes as $code){
if(str_pos($phonenumber, $code) === 0){
// phonenumber starts with areacode
$phoneWithoutCode = substr($phonenumber, strlen($code));
$preparedPhonenumber = $code.' '.$phoneWithoutCode;
break;
}
}
// if one of the areaCodes was 0349,
// the variable $preparedPhonenumber is now '0349 152023'
edit: you can shorten the amount of returned area codes from db by selecting only those that start with a certain string.
Let's assume the shortest area code in germany is 3 digits long (which i think is correct).
$threeDigits = substr($phonenumber,0,3);
$query = "SELECT * from areacodes
WHERE code like '".$threeDigits."%'
ORDER BY CHAR_LENGTH(code) DESC";
this will drastically shrink down the probable area codes array, therefore making the script faster.
edit 2: added order by clause in query so the above code will check for longer areacodes first. (the break; in the foreach loop is now obligatory!)
Hi Leonardo Gugliotti and Cashbee
i sort the areaCodes to get a better match. The php scripts works fine, but takes to long time to handle 5000 MySQL entries. Is it possible to make the foreach search directly in mySQL?
<?php
$sample_area_codes = array( '0350', '034', '034915', '03491', '0348', '0349', '03491', '034916', '034917',);
sort($sample_area_codes);
$phone_string = '0349152023';
foreach ($sample_area_codes as $code) {
$subString = substr($phone_string, 0, strlen($code));
if ($subString == $code) {
$phone = $subString." ".substr($phone_string, strlen($code));
}
}
if (!empty($phone)) {
echo $phone;
}
else {
echo "No AreaCode found.";
}
?>
Output: 034915 2023, which is correct
A single probe (assuming INDEX(area_code)):
SELECT ...
FROM AreaCodes
WHERE area_code < ?
ORDER BY area_code DESC
LIMIT 1;
(Where you bind the $phone_number as a string into the ?)
I think you'd better split your database into a tree, making a table for each digit.
So the third digit could refer to the second, the fourth to the third, and so on until you reach the maximum lenght of the prefix. The last table should include the name of the area.
Following your example, supposing that the maximum lenght of the area code was five digits, the fifth_digit_table should have at least four fields like these:
ID
IDref
Number
Name
10 records may have the same IDref, corresponding to the number "2" at the fourth position, linked to the previous "021" through the fourth_digit_table, the three_digit_table and so on; only one among these records, that with the Number field filled with "9", should have the Name "Haan"; the others, if there aren't any, should have the Name "Solingen".
I hope you will manage to speed up your script.
UPDATE: I think the cakePhp updateAll is the problem. If i uncomment the updateAll and pr the results i get in 1-2 seconds so many language Detections like in 5 minutes!!!! I only must update one row and can determine that row with author and title... is there a better and faster way???
I'm using detectlanguage.com in order to detect all english texts in my sql database. My Database consists of about 500.000 rows. I tried many things to detect the lang of all my texts faster. Now it will take many days... :/
i only send 20% of the text (look at my code)
i tried to copy my function and run the function many times. the copied code shows the function for all texts with a title starting with A
I only can run 6 functions at the same time... (localhost)... i tried a 7th function in a new tab, but
Waiting for available socket....
public function detectLanguageA()
{
set_time_limit(0);
ini_set('max_execution_time', 0);
$mydatas = $this->datas;
$alldatas = $mydatas->find('all')->where(['SUBSTRING(datas.title,1,1) =' => 'A'])->where(['datas.lang =' => '']);
foreach ($alldatas as $row) {
$text = $row->text;
$textLength = round(strlen($text)*0.2);
$text = substr($text,0,$ltextLength);
$title = $row->title;
$author = $row->author;
$languageCode = DetectLanguage::simpleDetect($text);
$mydatas->updateAll(
['lang' => $languageCode], // fields
['author' => $author,'textTitle' => $title]); // conditions*/
}
}
I hope some one has a idea for my problem... Now the language detection for all my texts will take more than one week :/ :/
My computer runs over 20 hours with only little interruptions... But i only detected the language of about 13.000 texts... And in my database are 500.000 texts...
Now i tried sending texts by batch, but its also to slow... I always send 20 texts in one Array and i think thats the maximum...
Is it possible that the cakePhp 3.X updateAll-function makes it so slowly?
THE PROBLEM WAS THE CAKEPHP updateAll
Now i'm using: http://book.cakephp.org/3.0/en/orm/saving-data.html#updating-data with a for loop and all is fast and good
use Cake\ORM\TableRegistry;
$articlesTable = TableRegistry::get('Articles');
for ($i = 1; $i < 460000; $i++) {
$oneArticle = $articlesTable->get($i);
$languageCode = DetectLanguage::simpleDetect($oneArticle->lyrics);
$oneArticle->lang = $languageCode;
$articlesTable->save($oneSong);
}
I'm working on building a social network from HTML, PHP, and a MySQL database, and I cant figure how to make this PHP code to work. Please remember, I'm totally not a pro at PHP.
What I'm working on now is a function, atreplace($text), that has the preg_replace() function in it to find the # mentions in a post caption, comment, or wherever else I use it, to make them into clickable links. Now that's pretty easy, except the #mentions in the strings it would process are user id numbers instead of the user name. For example, a string in a post caption for my social network would look like this:
"Went to Disneyland with my friends #214432 and #163728 today.". But what the post caption originally said was this: "Went to Disneyland with my friends #billysmith and #hi_im_kelly today.".
I wrote the script (not included because it doesn't have to do with the question) that processes the post data and inserts it into the MySQL database to replace the # mentions to be the user number id of the user that was mentioned instead of the user name in situations where people would have changed their username, then it wouldn't have to edit peoples posts where they mentioned someone that changed their username.
Problem
When it uses the atreplace() function to echo out the post caption when someone is viewing their feed, of course it will echo out "#214432" and "#163728" (from the example above) in the post caption instead of "#billysmith" and "#hi_im_kelly", which I would expect, because a different php script edited it and changed the usernames that were mentioned to be the user id, which is a number.
This is what I want
I want to write a function like the one I have below (get_username($id)). I only included get_username($id) to show what I want to do. I dont really know a lot about how to use the preg_replace() function yet.
As you can see, I tried passing the $1, which would be the user number id that I want it to replace with the username, with the get_username('$1) function, but it doesn't work. (I know I used that $1 wrong somehow). Although in the get_username() function, when I try returning the $id, it does output the user number id right. But when I used the $id in the mysql_query(), it didn't even show up.
In case you didn't understand all that, first, I want to put the string that has the # mentions in it, into the atreplace() function. This function will find all the # user mentions, remember, they are the users number id. Then, it will put each mention into the get_username() function. That function will get the username from a mysql database where the id equals the user number id that the atreplace() function found, then it will return the username, and in the end, will replace the # mention...
Can someone please show me how I could change the get_username() function to make it work? Or just write a new one.
<?php
function get_username($id){
$undata = mysql_query("select `username` from `accounts` where `id`='$id';");
$un = mysql_fetch_assoc($undata);
return "<a href='#'>#".$un['username']."</a>";
}
function atreplace($text){
$replaced = preg_replace('/\#(\w+)/', get_username('$1'), $text);
return($replaced);
}
?>
regarding mysql_* being deprecated and using prepared statements along with preg_replace_callback:
$pdo = new PDO('mysql:host=localhost;dbname=testdb;charset=utf8', 'localonly', 'localonly', array(
PDO::ATTR_EMULATE_PREPARES=>false,
PDO::MYSQL_ATTR_DIRECT_QUERY=>false,
PDO::ATTR_ERRMODE=>PDO::ERRMODE_EXCEPTION
));
...
function atreplace($text, $pdo) {
$stmt = $pdo->prepare('
SELECT
`username`
FROM
`so_profiles`
WHERE
`id`=?
');
$mesg = preg_replace_callback(
'/\#(\w+)/',
function ($match) use($stmt) {
$stmt->execute( array($match[1]) );
$users = $stmt->fetchAll(); // yes, there should be only one....
return empty($users) ? '[unknown user]' : $users[0];
},
$text
);
return $mesg;
}
In case anyone ever has that same problem, I got the answer thanks to #VolkerK
I just changed it to be one function:
function atreplace($text){
$mesg = preg_replace_callback('/\#(\w+)/',
function ($match){
$matchedit = preg_replace('/\#(\w+)/','$1', $match[0]);
$und = mysql_query("select `username` from `so_profiles` where `id`='".$matchedit."';");
$un = mysql_fetch_array($und);
return "<a>#".$un['username']."</a>";
},
$text);
return($mesg);
}
i get different string (like a, b, hospital, schools, jobs to work etc. ) as $divname from my db in my PHP section. i use this values as id in mydiv elemnents.
<div id="'.$divname.'"></div>
there is no problem for $divname = a, b, hospital or school but when it comes to jobs to work there is a huge problem cause my id gets spaces and it returns an error to me.
<div id="jobs to work"></div> //as evryone state spaces in id variable is an error.
now my question i need use this $divame variables in my id attribute. how can i do this? how can i delete those spaces or any more idea for using those in id attributes are welcome.
You may do two things:
Use a hashing function to create new id, which will be unqiue as long as your ids are unique as:
$newdivid = md5($olddivid);
You may write a function to remove spaces and combine such string items
function remove_spaces($str) {
$str_arr = explode(' ', $str);
$newstr = "";
foreach($str_arr as $sitem){
$newstr .= trim($sitem);
}
return $newstr;
}
Hope this solves your problem.
i found a php command in another Sof question for this purpose
$new_divname = str_replace(' ', '', $divname);
it deletes all spaces.. and of course my question gets a dublicate:
Currently I have a the following way of retrieving data from my DB:
$school->get('studentCount');
I required a shortcut to access these fields within the page and so came up with a format like this:
<p>Blah blah blah [[studentCount]]</p>
I have output buffering turned on but just need an easy way of replacing that key ('[[field-name]]') with its corresponding data from the DB.
If it was just one field I could do a str_replace on the output like this:
str_replace($output, '[[studentCount]]', $school->get('studentCount'))
Unfortunately that's not suitable. My ideal solution would grab whatever is between '[[' and ']]' and then run the 'get' method and replace the entire key ('[[...]]') with whatever is returned.
Well you could create two arrays, one with the field-name strings [[field-name]] and one with the responses $school->get('field-name'). Then throw those in str_replace as it supports arrays.
Example from PHP Manual:
$phrase = "You should eat fruits, vegetables, and fiber every day.";
$healthy = array("fruits", "vegetables", "fiber");
$yummy = array("pizza", "beer", "ice cream");
$newphrase = str_replace($healthy, $yummy, $phrase);
// Resulting String: "You should eat pizza, beer, and ice cream every day."
If you still wanted to implement your suggestion (finding all [[]]s and replacing them), I'll try to write up a quick function.
Edit: Here are two methods of doing it via your request:
$html = "Hello, [[FirstName]]! Welcome to [[SiteName]].";
$count = preg_match_all("/\[\[([\w]+)\]\]/", $html, $matches);
for ($x = 0; $x < $count; $x++)
$html = str_replace($matches[0][$x], $school->get($matches[1][$x]), $html);
Or using arrays:
$html = "Hello, [[FirstName]]! Welcome to [[SiteName]].";
$count = preg_match_all("/\[\[([\w]+)\]\]/", $html, $matches);
for ($x = 0; $x < $count; $x++)
$matches[1][$x] = $school->get($matches[1][$x]);
$html = str_replace($matches[0], $matches[1], $html);
I'm pretty sure this will work. :)
<?php
// $output contains the string
preg_match_all('/\[{2}([^\[]+)\]{2}/', $output, $matches);
$replaces = $matches['1'];
foreach($replaces as $replace) $str = str_replace('[['.$replace.']]', $school->get($replace), $output);
?>
You will need to use regex to find things inside of two [[ and ]] and take that an insert what is in between into your ->get() function.
Function would be preg_replace
http://us2.php.net/preg-replace
Assuming you can cache the result, a regex and file cache is a great method to do this. First you convert the file:
function cache_it($filename, $tablvar) {
$tmplt = file_get_contents($filename);
$tmplt = preg_replace('/\[\[(.+)\]\]/',
'<?php echo $' . $tablevar . '->get(\1);?>',
$tmplt);
file_put_contents($filename . '.php', $tmplt);
}
Then whenever you need to access the file.
function print_it($filename, $tablevar, $table) {
$_GLOBAL[$tablevar] = $table;
include $filename . '.php';
unset($_GLOBAL[$tablevar]);
}
You probably want to check that the cached file's create date is greater than the last modify date of the source file. Wrapping that and the two functions above in class helps avoid a lot of little pitfalls. But the general idea is sound. There are also some security issues with the cache file being a .php file that you would need to address.
I added this style template caching to the OSS CMS I work on. By caching the regex results, we sped up the original code by over 50%. The real benefit is the templates are PHP files. So anything that speeds the interpreting of PHP files (APC, eAccelerator, etc) speeds up your templates too.
You need to write a parser to look through the $output to find what is between your delimiters and then call a function if it is defined.
I assume you want to do it this way to save the calls until they are needed.
I wrote a template parser that effectively worked this way. Unfortunately it wasn't in PHP.
Thanks for the responses so far. I did think about using str/preg _replace with arrays but I wanted the key ('[[...]]') to directly tie in with the 'get' method, so it's fully expandable. I don't want to have to add to two different arrays every time a new field is added to the DB.
In JavaScript, for example, I would achieve it like this: (JavaScript allows you to pass an anonymous function as the 'replacement' parameter in its equivalent of preg_replace):
('this is the output blah blah [[studentCount]]').replace(/\[{2}([^\[]+)\]{2}/g, function($0, $1) {
get($1);
})