I have a table in MySQL with approx. 20 million rows.
id | word_eng | word_indic
I have to translate the english word (word_eng) into indian language (word_indic) using google translate api.
I have written PHP code which spawns multiple curl requests and fetches data from API and updates it into the table. But this process is quite slow, about 100 to 200 words per second.
I am using RollingCurl for multi curl.
Whats the best way to make it as fast as possible?
Below is my code. I am running this as a cron job.
<?php
include_once('db.php');
include_once('functions.php');
include_once('rolling-curl-master/RollingCurl.php');
$table = $argv[1];
$q = "SELECT * from $table where word_indic is null limit 500000";
$result = $conn->query($q); $n = 0;
$urls = array();
while ($row = $result->fetch_assoc())
{
$id = $row['id'];
$word = rawurlencode(getName($row['name_eng']));
//getName is a simple function which does some trimming and cleaning up of string
$url = 'https://www.google.com/inputtools/request?text='.rawurlencode($word).'&ime=transliteration_en_te&id='.rawurlencode($id);
array_push($urls, $url);
}
//print_r($urls);
unset($url);
$rc = new RollingCurl("request_callback");
// the window size determines how many simultaneous requests to allow.
$rc->window_size = 300;
foreach ($urls as $url)
{
// add each request to the RollingCurl object
$request = new RollingCurlRequest($url);
$rc->add($request);
}
$rc->execute();
function request_callback($response, $info)
{
// parse the page title out of the returned HTML
if (preg_match("~<title>(.*?)</title>~i", $response, $out)) {
$title = $out[1];
}
//echo "<b>$title</b><br />";
//print_r($info);
$parts = parse_url($info['url']);
parse_str($parts['query'], $query);
$id = $query['id'];
$text = $query['text'];
//echo "<hr>";
$trans = json_decode($response)[1][0][1][0];
global $conn; global $table; global $urls; global $n;
if ($trans != '' and !preg_match('/[a-z]/', $trans))
{
$conn->query("update $table set word_indic='$trans' where id='$id'"); $n++;
}
}
?>
Related
Is there a way I can Array_chunk mysqli results, I am looping messages from a table and later pass the values into a method "Sms" The method will create a List of Sms objects which I pass through a function SendBatchSMS. my API end points can only allow 100 call per request.
I have tried array chunking the list into "$sms" which seams to work well when I print_r($sms), but when echo the response, it returns only 48/249 responses regardless of the size specified in the array_chunk function. My question is, is there a better option to achieve this, something like array_chunking the mysqli results instead of the array list?
$query_sch = "SELECT * FROM ct_queue";
$sch_result = mysqli_query($mysqli, $query_sch);
$rows[] = mysqli_fetch_array($sch_result);
$count = mysqli_num_rows($sch_result);
foreach($sch_result as $value)
{
$phone = $value['phone'];
$sender = $value['sender'];
$message = $value['message'];
$user_id = $value['user_id'];
$link_id = NULL;
$correlator = 'correlator_string';
$endpoint = 'example.com';
$token = "token_string";
// $list = array();
$version = "v1"; //DONT change unless you are using a different version
$instance = new BonTech($token, $version);
$list[] = new Sms($sender, $phone, $message, $correlator, null, $endpoint);
}
$row_chunks = array_chunk($list, 100);
foreach ($row_chunks as $chunk){
$sms = array();
////////here we have 100 messages on each chunk
///////Loop through the messages in side the chunk
foreach ($chunk as $row) {
$sms[] = ($row);
}
// print_r($sms);
}
$response = call_user_func_array(array($instance, "sendBatchSMS"), $sms);
$response = json_encode($response, true);
$results = json_decode($response, true);
print_r($response);
You're using $sms after the foreach loop is done. So it will only contain the last chunk. You need to use it inside the loop.
There's also no need to use a loop to copy $chunk to $sms.
You're also skipping the first row of results because of your call to mysqli_fetch_array($sch_result) before the first foreach loop.
$instance doesn't seem to be dependent on $value, so it shouldn't be in the foreach loop.
$query_sch = "SELECT * FROM ct_queue";
$sch_result = mysqli_query($mysqli, $query_sch);
$list = array();
foreach($sch_result as $value)
{
$phone = $value['phone'];
$sender = $value['sender'];
$message = $value['message'];
$user_id = $value['user_id'];
$link_id = NULL;
$correlator = 'correlator_string';
$endpoint = 'example.com';
$list[] = new Sms($sender, $phone, $message, $correlator, null, $endpoint);
}
$token = "token_string";
$version = "v1"; //DONT change unless you are using a different version
$instance = new BonTech($token, $version);
$row_chunks = array_chunk($list, 100);
foreach ($row_chunks as $sms){
$response = call_user_func_array(array($instance, "sendBatchSMS"), $sms);
print_r($response);
}
I have a URL string of images divided by "|" and I would like a php function that reading the string separate the images and divide them with "," to use a wordpress gallery component
http://xxxxxxxx/xxxxxxx/wp-content/uploads/2018/02/large-IMG_5367.jpg|http://xxxxxxxx/xxxxxxx/wp-content/uploads/2018/02/large-IMG_5376.jpg|http://xxxxxxxx/xxxxxxx/wp-content/uploads/2018/02/large-IMG_6324.jpg
I tried to create a php shortcode called get id from string
/* ---------------------------------------------------------------------------
* Shortcode | get_id_from_string
* --------------------------------------------------------------------------- */
add_shortcode('get_id_from_string', 'function_get_id_from_string');
function function_get_id_from_string($atts) {
global $wpdb;
$return_value = '';
$url_array = explode('|', $atts['urls']);
foreach ($url_array as &$url) {
$return_value .= $wpdb->get_col($wpdb->prepare("SELECT ID FROM $wpdb->posts WHERE guid='%s';", $url))[0] . ',';
}
return rtrim($return_value, ',');
}
But it does not work, has anyone already done something like that?
Thanks in advance
Try below cod,
If URL found in database it will return ID and add in $return_value variable.
add_shortcode('get_id_from_string', 'function_get_id_from_string');
function function_get_id_from_string($atts)
{
global $wpdb;
//$imagestr = "http://xxxxxxxx/xxxxxxx/wp-content/uploads/2018/02/large-IMG_5367.jpg|http://xxxxxxxx/xxxxxxx/wp-content/uploads/2018/02/large-IMG_5376.jpg|http://xxxxxxxx/xxxxxxx/wp-content/uploads/2018/02/large-IMG_6324.jpg";
$imagestr = $atts['urls'];
$url_array = explode("|", $imagestr);
$return_value = [];
foreach ($url_array as $url) {
$query = $wpdb->prepare("SELECT ID FROM $wpdb->posts WHERE guid='%s'", $url);
$result = $wpdb->get_col($wpdb->prepare($query, $url), ARRAY_A);
if (!empty($result))
$return_value[] = $result[0];
}
$images_ids = implode(",", $return_value);
return ($images_ids);
}
I'm working on a project in which I pull various statistics about the NHL and inserting them into an SQL table. Presently, I'm working on the scraping phase, and have found an XML parser that I've implemented, but I cannot for the life of me figure out how to pull information from it. The table can be found here -> http://www.tsn.ca/datafiles/XML/NHL/standings.xml.
The parser supposedly generates a multi-dimmensional array, and I'm simply trying to pull all the stats from the "info-teams" section, but I have no idea how to pull that information from the array. How would I go about pulling the number of wins Montreal has? (Solely as an example for the rest of the stats)
This is what the page currently looks like -> http://mattegener.me/school/standings.php
here's the code:
<?php
$strYourXML = "http://www.tsn.ca/datafiles/XML/NHL/standings.xml";
$fh = fopen($strYourXML, 'r');
$dummy = fgets($fh);
$contents = '';
while ($line = fgets($fh)) $contents.=$line;
fclose($fh);
$objXML = new xml2Array();
$arrOutput = $objXML->parse($contents);
print_r($arrOutput[0]); //This print outs the array.
class xml2Array {
var $arrOutput = array();
var $resParser;
var $strXmlData;
function parse($strInputXML) {
$this->resParser = xml_parser_create ();
xml_set_object($this->resParser,$this);
xml_set_element_handler($this->resParser, "tagOpen", "tagClosed");
xml_set_character_data_handler($this->resParser, "tagData");
$this->strXmlData = xml_parse($this->resParser,$strInputXML );
if(!$this->strXmlData) {
die(sprintf("XML error: %s at line %d",
xml_error_string(xml_get_error_code($this->resParser)),
xml_get_current_line_number($this->resParser)));
}
xml_parser_free($this->resParser);
return $this->arrOutput;
}
function tagOpen($parser, $name, $attrs) {
$tag=array("name"=>$name,"attrs"=>$attrs);
array_push($this->arrOutput,$tag);
}
function tagData($parser, $tagData) {
if(trim($tagData)) {
if(isset($this->arrOutput[count($this->arrOutput)-1]['tagData'])) {
$this->arrOutput[count($this->arrOutput)-1]['tagData'] .= $tagData;
}
else {
$this->arrOutput[count($this->arrOutput)-1]['tagData'] = $tagData;
}
}
}
function tagClosed($parser, $name) {
$this->arrOutput[count($this->arrOutput)-2]['children'][] = $this->arrOutput[count($this- >arrOutput)-1];
array_pop($this->arrOutput);
}
}
?>
add this search function to your class and play with this code
$objXML = new xml2Array();
$arrOutput = $objXML->parse($contents);
// first param is always 0
// second is 'children' unless you need info like last updated date
// third is which statistics category you want for example
// 6 => the array you want that has wins and losses
print_r($arrOutput[0]['children'][6]);
//using the search function if key NAME is Montreal in the whole array
//result will be montreals array
$search_result = $objXML->search($arrOutput, 'NAME', 'Montreal');
//first param is always 0
//second is key name
echo $search_result[0]['WINS'];
function search($array, $key, $value)
{
$results = array();
if (is_array($array))
{
if (isset($array[$key]) && $array[$key] == $value)
$results[] = $array;
foreach ($array as $subarray)
$results = array_merge($results, $this->search($subarray, $key, $value));
}
return $results;
}
Beware
this search function is case sensitive it needs modifications like match to
a percentage the key or value changing capital M in montreal to lowercase will be empty
Here is the code I sent you working in action. Pulling the data from the same link you are using also
http://sjsharktank.com/standings.php
I have actually used the same exact XML file for my own school project. I used DOM Document. The foreach loop would get the value of each attribute of team-standing and store the values. The code will clear the contents of the table standings and then re-insert the data. I guess you could do an update statement, but this assumes you never did any data entry into the table.
try {
$db = new PDO('sqlite:../../SharksDB/SharksDB');
$db->setAttribute(PDO::ATTR_ERRMODE,PDO::ERRMODE_EXCEPTION);
} catch (Exception $e) {
echo "Error: Could not connect to database. Please try again later.";
exit;
}
$query = "DELETE FROM standings";
$result = $db->query($query);
$xmlDoc = new DOMDocument();
$xmlDoc->load('http://www.tsn.ca/datafiles/XML/NHL/standings.xml');
$searchNode = $xmlDoc->getElementsByTagName( "team-standing" );
foreach ($searchNode as $searchNode) {
$teamID = $searchNode->getAttribute('id');
$name = $searchNode->getAttribute('name');
$wins = $searchNode->getAttribute('wins');
$losses = $searchNode->getAttribute('losses');
$ot = $searchNode->getAttribute('overtime');
$points = $searchNode->getAttribute('points');
$goalsFor = $searchNode->getAttribute('goalsFor');
$goalsAgainst = $searchNode->getAttribute('goalsAgainst');
$confID = $searchNode->getAttribute('conf-id');
$divID = $searchNode->getAttribute('division-id');
$query = "INSERT INTO standings ('teamid','confid','divid','name','wins','losses','otl','pts','gf','ga')
VALUES ('$teamID','$confID','$divID','$name','$wins','$losses','$ot','$points','$goalsFor','$goalsAgainst')";
$result= $db->query($query);
}
I am trying to get specific data from a while loop and loop it x number of times.
I'm selecting this data:
$r=mysql_query("SELECT ac6, ac5, ac4, ac3, ac2, ac1, ac0 FROM advertisements WHERE token = '".$_GET['token']."'");
while ($adData = mysql_fetch_array($r, MYSQL_NUM))
{
$data = $adData;
$ac0 = $data['ac0'];
$ac1 = $data['ac0'];
print $ac0;
print $ac1;
}
This doesn't work. Nothing gets printed out.
What I want to do is to get ac6 to ac0 value for that specific advertisement (where token).
How can I do that?
Change your numeric fetch to an associative one, then add a foreach loop to process the result.
$r = mysql_query("SELECT ac6, ac5, ac4, ac3, ac2, ac1, ac0
FROM advertisements WHERE token = '" . $_GET['token'] . "'");
while ($adData = mysql_fetch_assoc($r))
{
foreach ($adData as $key => $value)
{
$nubmer = (int)substr($key, 2);
print $value; // or whatever you actually want to do
}
}
Also I hope you're validating $_GET['token'] against possible mischief in your code.
You can create an arrays to add the values from the result query
1st is to create an array:
$advert_AC0 = array();
$advert_AC1 = array();
$advert_AC2 = array();
$advert_AC3 = array();
$advert_AC4 = array();
$advert_AC5 = array();
$advert_AC6 = array();
Now, to add content to array
$r = mysql_query("SELECT ac6, ac5, ac4, ac3, ac2, ac1, ac0
FROM advertisements WHERE token = '" . $_GET['token'] . "'");
if(mysql_num_rows($r)){ //check 1st if there is num of rows from the result
while ($adData = mysql_fetch_assoc($r))
{
array_push($advert_AC0, $dData['ac0']);
array_push($advert_AC1, $dData['ac1']);
array_push($advert_AC2, $dData['ac2']);
array_push($advert_AC3, $dData['ac3']);
array_push($advert_AC4, $dData['ac4']);
array_push($advert_AC5, $dData['ac5']);
array_push($advert_AC6, $dData['ac6']);
}
}else{
echo "NO RESULT.";
}
to call for the array values, 1 by 1
$count_array = count($advert_AC0);
$i = $count_array;
while(1 <= $i){
echo "AC0: $advert_AC0[$i]<br>";
echo "AC1: $advert_AC1[$i]<br>";
echo "AC2: $advert_AC2[$i]<br>";
echo "AC3: $advert_AC3[$i]<br>";
echo "AC4: $advert_AC4[$i]<br>";
echo "AC5: $advert_AC5[$i]<br>";
echo "AC6: $advert_AC6[$i]<br>";
$i++;
}
I don't know if my answer solved your question, please comment if not.
Can't find quite the right answer so hope someone can help. Basically want to create an array and then return the results from a search e.g.
$tsql = "SELECT date, staffid, ID,status, eventid, auditid from maincalendar";
$params = array();
$options = array( "Scrollable" => SQLSRV_CURSOR_KEYSET );
$stmt = sqlsrv_query( $conn, $tsql , $params, $options);
$calarray=array();
while($row = sqlsrv_fetch_array($stmt)) {
$rowresult = array();
$rowresult["status"] = $row['status'];
$rowresult["eventid"] = $row['eventid'];
$rowresult["caldate"] = $row['date'];
$rowresult["staffid"] = $row['staffid'];
$rowresult["ID"] = $row['ID'];
$rowresult["auditid"] = $row['auditid'];
$calarray[] = $rowresult;
}
I would then like to search for values matching 'caldate' and 'staffid' and return the associated entry in $calarray
I suggest the following,
Fetch all data needed for the current month you are showing, using col BETWEEN x AND y
Add them to a array in PHP, with staffid and caldate as key
Something like so;
$calarray[$row['staffid'] . '-' . $row['date']][] = $row;
Not sure if a single staffid/date combination can have one or more events per day, if not you can remove the []
To check if we have information for a specific staffid/date combination, use isset
if (isset($calarray[$staffid . '-' . $mydate]) { ... }
Add indexes to the fields you're going to query, and then move the search to the sql query. You can also make simple filtering inside the loop. So you'll be populating several arrays instead of one, based on the search you need.
try this:
$matches = array();
$caldate = //your desired date;
$staffid = //your desired id;
foreach($calarray as $k => $v){
if(in_array($caldate, $v['caldate']) && in_array($staffid, $v['staffid'])){
$matches[] = $calarray[$k];
}
}
$matches should be and array with all the results you wanted.
also:
while($row = sqlsrv_fetch_array($stmt)) {
$rowresult = array();
$rowresult["status"] = $row['status'];
$rowresult["eventid"] = $row['eventid'];
$rowresult["caldate"] = $row['date'];
$rowresult["staffid"] = $row['staffid'];
$rowresult["ID"] = $row['ID'];
$rowresult["auditid"] = $row['auditid'];
$calarray[] = $rowresult;
}
can be shortened into:
while($row = sqlsrv_fetch_array($stmt)) {
$calarray[] = $row;
}
Maybe this code snipplet solves your problem.
I am not a PHP programmer, so no warrenty.
function searchInArray($array, $keyword) {
for($i=0;$i<array.length();$i++) {
if(stristr($array[$i], $keyword) === FALSE) {
return "Found ".$keyword." in array[".$i."]";
}
}
}