I have a two PHP scripts that are loading many variable resources from APIs, causing the response times to as long as 2.2 seconds to 4 seconds. Any suggestions on how to decrease response times and increase efficiency would be very appreciated?
FIRST SCRIPT
require('path/to/local/API_2');
//Check if user has put a query and that it's not empty
if (isset($_GET['query']) && !empty($_GET['query'])) {
//$query is user input
$query = str_replace(" ", "+", $_GET['query']);
$query = addslashes($query);
//HTTP Request to API_1
//Based on $query
//Max Variable is ammount of results I want to get back in JSON format
$varlist = file_get_contents("http://ADRESS_OF_API_1.com?$query&max=10");
//Convert JSON to Array()
$varlist = json_decode($varlist, true);
//Initializing connection to API_2
$myAPIKey = 'KEY';
$client = new APIClient($myAPIKey, 'http://ADRESS_OF_API_2.com');
$Api = new API_FUNCTION($client);
$queries = 7;
//Go through $varlist and get data for each element in array then use it in HTML
//Proccess all 8 results from $varlist array()
for ($i = 0; $i <= $queries; ++$i) {
//Get info from API based on ID included in first API data
//I don't use all info, but I can't control what I get back.
$ALL_INFO = $Api->GET_FUNCTION_1($varlist[$i]['id']);
//Seperate $ALL_INFO into info I use
$varlist[$i]['INFO_1'] = $ALL_INFO['PATH_TO_INFO_1'];
$varlist[$i]['INFO_2'] = $ALL_INFO['PATH_TO_INFO_2'];
//Check if info exists
if($varlist[$i]['INFO_1']) {
//Concatenate information into HTML
$result.='
<div class="result">
<h3>'.$varlist[$i]['id'].'</h3>
<p>'.$varlist[$i]['INFO_1'].'</p>
<p>'.$varlist[$i]['INFO_2'].'</p>
</div>';
} else {
//In case of no result for specific Info ID increase
//Allows for 3 empty responses
++$queries;
}
}
} else {
//If user didn't enter a query, relocates them back to main page to enter one.
header("Location: http://websitename.com");
die();
}`
NOTE: $result equals HTML information from each time arround the loop.
NOTE: Almost all time is spent in the for ($i = 0; $i <= 7; ++$i)
loop.
SECOND SCRIPT
//Same API as before
require('path/to/local/API_2');
//Check if query is set and not empty
if (isset($_GET['query']) && !empty($_GET['query'])) {
//$query is specific $varlist[$i]['id'] for more information on that data
$query['id'] = str_replace(" ", "+", $_GET['query']);
$query['id'] = addslashes($query['id']);
//Initializing connection to only API used in this script
$myAPIKey = 'KEY';
$client = new APIClient($myAPIKey, 'http://ADRESS_OF_API_2.com');
$Api = new API_FUNCTION($client);
$ALL_INFO_1 = $Api->GET_FUNCTION_1($query['id']);
$query['INFO_ADRESS_1.1'] = $ALL_INFO_1['INFO_ADRESS_1'];
$query['INFO_ADRESS_1.2'] = $ALL_INFO_2['INFO_ADRESS_2'];
$ALL_INFO_2 = $Api->GET_FUNCTION_2($query['id']);
$query['INFO_ADRESS_2.1'] = $ALL_INFO_3['INFO_ADRESS_3'];
$ALL_INFO_3 = $Api->GET_FUNCTION_3($query['id']);
$query['INFO_ADRESS_3.1'] = $ALL_INFO_4['INFO_ADRESS_4'];
$ALL_INFO_4 = $Api->GET_FUNCTION_4($query['id']);
$query['INFO_ADRESS_4.1'] = $ALL_INFO_5['INFO_ADRESS_5'];
$query['INFO_ADRESS_4.2'] = $ALL_INFO_6['INFO_ADRESS_6'];
$ALL_INFO_5 = $Api->GET_FUNCTION_5($query['id']);
$query['INFO_ADRESS_5.1'] = $ALL_INFO_7['INFO_ADRESS_7'];
}
$result = All of the $query data from the API;
} else {
//If no query relocates them back to first PHP script page to enter one.
header("Location: http://websitename.com/search");
die();
}`
NOTE: Similiarly to the first script, most time is spent getting info
from the secondary API.
NOTE: In the second script, the first API is replaced by a single
specific variable from the first script page,so $varlist[$i]['id'] =
$query['id'].
NOTE: Again, $result is the HTML data.
You could also move the API calls out from your normal page load. Respond to the user with a generic page to show something is happening and then make an ajax request to query the APIs and respond with data. There really is no way to speed up an individual external request. Your best bet is to:
try to minimize the number of requests (even if it means you request a little more data once then filter out on your side vs sending multiple requests for a small subset of data).
cache any remaining requests and pull from cache.
respond with a small page to let the user know something is happening and make separate ajax requests for the queried data.
Related
I am building a website which scrapes data from another website,stores it in a database and shows it in the form of a table. Everything works fine as long as the number of rows are less (around 100), but when the data set increases, say 300 rows or more the data gets stored in the database (phpmyadmin) but nothing shows on the screen and the site just keeps loading. Below is a section of the php script i am running:
<?php
// configuration
require("../includes/helpers.php");
// initializing current page and number of pages
$page = 0;
$pages = 1;
// scrape data from each page
while($pages--)
{
// next page
$page++;
// scrape data from shiksha.com
$string = #file_get_contents("http://www.shiksha.com/b-tech/colleges/b-tech-colleges-".urlencode($_POST["city"])."-{$page}");
if($string === false)
apologize("Please enter a valid city name");
if($page === 1)
{
// counting total number of pages
preg_match_all('/class=" linkpagination">/',$string,$result);
$pages = sizeof($result[0]);
}
// passing the string for scraping data and storing in database
get_college_info($string,$page);
// delay for 2s
sleep(2);
}
// querying the infrastructure table for facilities of all colleges
$infra = query("SELECT college_id,facilities FROM infrastructure ");
// preparing query and selecting data from table college_info
$result = query("SELECT * FROM college_info");
// render(output) results
render("result.php",["title" => "result","infra" => $infra,"result" => $result]);
}
}?>
interestingly, if i already have the data stored in my db and I just retrieve and print it , everything works fine and all the data ,however large it is,gets printed. I have no clue whats the problem.
PS : I have already tried set_time_limit().
you are creating an infinite loop. so to fix the issue change the criteria for your while loop to the below.
while($page<$pages)
{
//your same code here
}
This question already has answers here:
Can I mix MySQL APIs in PHP?
(4 answers)
Closed 6 years ago.
I am attempting to implement a click count system. I am using the following code in this link Click here to see code, but changing it to modern standards. Initially I received errors for the msqli_real_escape_ string, but I believed I resolved it(no errors). Now, I am not receiving any errors at all, but the query is not sending into my database. I am using ini_set('display_errors', 1);
error_reporting(E_ALL); for error checking. Also I have my $con and session in and ini file that I call, so the session and connection are not issues.
Does anyone see what I am doing wrong or is there a good way I can check to see what isn't working?
//create current page constant
$curPage = mysqli_real_escape_string($con,htmlspecialchars($_SERVER['PHP_SELF']));
//set number of clicks variable to 0
$clicks = 0;
//do not recount if page currently loaded
if($_SESSION['page'] != $curPage) {
//set current page as session variable
$_SESSION['page'] = $curPage;
$click_sql = "
SELECT *
FROM click_count
WHERE page_url = ?
";
if (!$click_stmt = $con->prepare($click_sql)) {
$click_stmt->bind_param("s", $curPage);
$click_stmt->execute();
$num_rows = $click_stmt->fetchColumn();
if (!$click_stmt->errno) {
// Handle error here
}
$stmt->bind_result($click_id, $page_url, $page_count);
} elseif ($num_rows == 0) {
//try to create new record and set count for new page to 1
//output error message if problem encountered
$click_insert_stmt = "
INSERT INTO click_count
(page_url, page_count)
VALUES(?, ?)";
if(!$click_stmt = $con->prepare($click_insert_stmt)) {
$click_insert_stmt->execute(array('$curPage',1));
echo "Could not create new click counter.";
}
else {
$clicks= 1;
}
} else {
//get number of clicks for page and add 1 fetch(PDO::FETCH_BOTH)
while($click_row = $click_insert_stmt->fetch(PDO::FETCH_BOTH)) {
$clicks = $row['page_count'] + 1;
//update click count in database;
//report error if not updated
$click_update_stmt = "
UPDATE click_count
SET page_count = ?
WHERE page_url = ?
";
if(!$click_stmt = $con->prepare("$click_update_stmt")) {
$click_update_stmt->execute(array('$clicks', '$curPage'));
echo "Could not save new click count for this page.";
}
}
}
}
Edit: New Updated Code
// ********Page count************
//create current page constant
$curPage = mysqli_real_escape_string($con,($_SERVER['PHP_SELF']));
//set number of clicks variable to 0
$clicks = 0;
//do not recount if page currently loaded
if($_SESSION['page'] != $curPage) {
//set current page as session variable
$_SESSION['page'] = $curPage;
$click_sql = "
SELECT *
FROM click_count
WHERE page_url = ?
";
if (!$click_stmt = $con->prepare($click_sql)) {
$click_stmt->bind_param("s", $_SERVER['PHP_SELF']);
$click_stmt->execute();
$num_rows = $click_stmt->fetchColumn();
if (!$click_stmt->errno) {
// Handle error here
}
$stmt->bind_result($click_id, $page_url, $page_count);
} elseif ($num_rows == 0) {
//try to create new record and set count for new page to 1
//output error message if problem encountered
$click_insert_stmt = "
INSERT INTO click_count
(page_url, page_count)
VALUES(?, ?)";
if(!$click_stmt = $con->prepare($click_insert_stmt)) {
$click_insert_stmt->execute(array($curPage,1));
echo "Could not create new click counter.";
}
else {
$clicks= 1;
}
} else {
//get number of clicks for page and add 1 fetch(PDO::FETCH_BOTH)
while($click_row = $click_insert_stmt->fetch(PDO::FETCH_BOTH)) {
$clicks = $row['page_count'] + 1;
//update click count in database;
//report error if not updated
$click_update_stmt = "
UPDATE click_count
SET page_count=page_count+1
WHERE page_url = ?
";
if(!$click_stmt = $con->prepare("$click_update_stmt")) {
$click_update_stmt->execute(array($curPage));
echo "Could not save new click count for this page.";
}
}
}
}
It looks like you're doing a lot of stuff like this:
$click_update_stmt->execute(array('$clicks', '$curPage'));
I'm not sure where you picked up this habit of quoting variables as strings, but you need to drop it. '$x' and $x are two hugely different things. In the first case it's literally '$x' and in the second case it's whatever the $x variable happens to represent.
Fix it like this:
$click_update_stmt->execute(array($clicks, $curPage));
Also since you're using prepared statements, which by the way is great, you do not need to and should not manually escape your values. Applying them to placeholders with bind_param is the safe way of doing it. Doing any other escaping mangles the data.
Just bind directly to the source:
$click_stmt->bind_param("s", $_SERVER['PHP_SELF']);
Don't arbitrarily run things like htmlspecialchars on input out of paranoia or because you're doing cargo-cult programming and you saw it done in a YouTube tutorial somewhere. That function is intended to be used to display values only, not store them. Data in your database should be as raw as possible.
There's a lot of problems with this code, and one of them that has me confused is why there's so much code. Remember SELECT * and then binding results to arbitrary variables is trouble, your schema might change and then your code is out of sync. Whenever possible fetch rows as an associative array if doing this, then all you have to worry about is renamed ore removed columns.
The biggest problem is this is subject to race conditions because it doesn't use an atomic increment. When writing counters, always do your updates as operations that are a single statement:
UPDATE click_count SET page_count=page_count+1 WHERE page_url=?
Your approach of reading the count, incrementing it, and then writing it back into the database means that you're inviting problems if another operation runs concurrently, something very likely on click-counter code.
I installed fork on my Ubuntu Server (using PHP-Apache-Codeigniter), and checked if it's working using var_dump (extension_loaded('pcntl')); and got a "true" output (How to check PCNTL module exists).
I have this code:
public function add_keyword() {
$keyword_p = $this->input->post('key_word');
$prod = $this->input->post('prod_name');
$prod = $this->kas_model->search_prod_name($prod);
$prod = $prod[0]->prod_id;
$country = $this->input->post('key_country');
$keyword = explode(", ", $keyword_p);
var_dump($keyword);
$keyword_count = count($keyword);
echo "the keyword count: $keyword_count";
// Create fork
$pid = pcntl_fork();
if(!$pid){
for ($i=0; $i < $keyword_count ; $i++) {
// Inserts the inputs to the "keywords" table
$this->kas_model->insert_keyword($keyword[$i], $prod, $country);
// Gets relevant IDs for the inserted prod and keyword
$last_inserted_key = $this->kas_model->get_last_rec('keywords');
$keyword_id = $last_inserted_key[0]->key_id;
$prod_id = $last_inserted_key[0]->key_prod;
$prod_id_query = $this->kas_model->get_prod_row_by_id($prod_id);
$prod_id_a = $prod_id_query[0]->prod_a_id;
$prod_id_b = $prod_id_query[0]->prod_b_id;
// Run the keyword query (on API) for today on each one of the keys and insert to DB aslong that the ID isn't 0.
if ( ($prod_id_a != 0) || ( !empty($prod_id_a) ) ) {
$a_tdr = $this->get_var1_a_by_id_and_kw( $prod_id_a, $keyword[$i], $country);
} else {
$a_tdr['var1'] = 0;
$a_tdr['var2'] = 0;
$a_tdr['var3'] = 0;
}
if ( ($prod_id_b != 0) || ( !empty($prod_id_b) ) ) {
$b_tdr = $this->get_var1_b_by_id_and_kw($prod_id_b, $keyword[$i], $country);
} else {
$b_tdr['var1'] = 0;
$b_tdr['var2'] = 0;
$b_tdr['var3'] = 0;
}
$this->kas_model->insert_new_key_to_db($keyword_id, $a_tdr['var1'], $b_tdr['var1'], $a_tdr['var2'], $b_tdr['var2'], $a_tdr['var3'], $b_tdr['var3']);
}
exit($i);
}
// we are the parent (main), check child's (optional)
while(pcntl_waitpid(0, $status) != -1){
$status = pcntl_wexitstatus($status);
// echo "Child $status completed\n";
redirect('main/kas');
}
redirect('main/kas');
}
What the function does?
This function gets 1 or more keyword/s, a country var, and a product ID, and runs a query on an external slow API getting variables (runs other functions from within that same controller), and adds them to the database.
Problem: When running this function, and if I insert a lot of keywords, the page loads, and loads, and loads, for a long time, until it's done, and only then - I can continue browsing my website. So I was told to fork it since it's just sending a request to process it in the background, so whenever clicking the submit button, I get redirected to "main/kas".
Currently: I don't get redirected but the function runs without any errors.
I was told that it suppose to work but it's not - so I'm guessing I am doing something wrong within the code (?), or something else isn't working from within the server (???). This is my first time working with fork so I don't know a lot of how to operate with in (in syntax or from within the server).
Can you please help me debug the problem?
http://www.electrictoolbox.com/mysql-connection-php-fork/
Reason for the error The parent and child processes all share the same
database connection. When the first child process exits it will
disconnect from the database, which means the same connection all
processes are using will be disconnected, causing any further queries
to fail.
The solution The solution is to disconnect from the database before
forking the sub processes and then establish a new connection in each
process. The fourth parameter also should be passed to the
mysql_connect function as "true" to ensure a new link is established;
the default is to share an existing connection is the login details
are the same.
The question is!
Is that efficient to connect to the server in the child and if there are any other alternative ways to do this better.
I am writing a custom Moodle (2.6) web service for taking up the moodle quiz, that requires to create an attempt or start an attempt. I written the API to start an attempt as well as get the questions for a particular quiz. However I m not able to figure out how to save the User responses for the question attempt. (Question attempt step data). Will appreciate if anyone can help me.
//Fetch all the user attempts
$attempts = quiz_get_user_attempts($quizObj->get_quizid(), 3, 'all', true);
//Get the last attempt
$lastattempt = end($attempts);
// Delete any previous preview attempts belonging to this user.
quiz_delete_previews($quizObj->get_quiz(), $USER->id);
//fetch the quiz usage object
$quba = question_engine::make_questions_usage_by_activity('mod_quiz', $quizObj->get_context());
$quba->set_preferred_behaviour($quizObj->get_quiz()->preferredbehaviour);
// Create the new attempt and initialize the question sessions
$timenow = time(); // Update time now, in case the server is running really slowly.
$attempt = quiz_create_attempt($quizObj, $attemptnumber, $lastattempt, $timenow, $quizObj->is_preview_user());
if ($lastAttemptStatus == quiz_attempt::FINISHED) {
$attemptnumber = $lastattempt->attempt + 1;
$attempt = quiz_start_new_attempt($quizObj, $quba, $attempt, $attemptnumber, $timenow);
} elseif ($lastAttemptStatus == quiz_attempt::IN_PROGRESS) {
$attempt = quiz_start_attempt_built_on_last($quba, $attempt, $lastattempt);
}
// It is here Lets say I would like to hardcode (get param) the responses from the user for a quiz for a specific question must go.. and I have add these responses as attempt step data..
$transaction = $DB->start_delegated_transaction();
$attempt = quiz_attempt_save_started($quizObj, $quba, $attempt);
quiz_fire_attempt_started_event($attempt, $quizObj);
$transaction->allow_commit();
$lastattempt = end($attempts);
For saving quiz attempts you need to call mod_quiz_save_attempt API
with question and answers response in parameters.
For eg:
http://**yourdomain.com**/webservice/rest/server.php?wstoken=**yourToken**&wsfunction=mod_quiz_save_attempt&moodlewsrestformat=json&attemptid=4545&data[0][name]=slots&data[0][value]=2&data[1][name]=q87124:2_:sequencecheck&data[1][value]=1&data[2][name]=q87124:2_answer&data[2][value]=2
For more details refer this link
I am managing a pool of stateless session for a web service between users.
So when a user request web service he start session and response timeout is 5 sec, so he can hold session for 5 sec max. second user comes in and system check if there is available session then use it.
Now i have a problem here. when let say there is a session available, user A comes, system check if it was used more than 5 sec ago, given to user A, at the same time another user hit and system check if session used more than 4 sec ago, assigned to user B.
Now both user using same session and system fails on one.
I have tried select of update command as well to lock it.
I have tried update last used time as soon as it is selected by first user, but this didn't work (i think second user hitting system as the same time)
can someone advice on it.
code: check for session from db if available then pick it or insert a new
//get 25 sessions from database order by lastqueryDate
$session = $sessionObj->select('session', '*', '', 'LastQueryDate DESC', '25');
$available_session = array();
//if sessions available, get rows from getResult
if ($session) {
$session_Data = $sessionObj->getResult();
//now get session which is sitting there more than response time out
$available_session = $sessionObj->getAvailableSession($session_Data);
}
//if there is any, use it. otherwise create new session and save in database
if (!$available_session) {
$auth->securityAuthenticate();
$header = $auth->getHeaders();
$sequence = (int) $header['Session']->SequenceNumber + 1;
$values[] = $header['Session']->SessionId;
$values [] = $sequence;
$values [] = $header['Session']->SecurityToken;
$rows = "SessionID,SequenceNo,Security_token";
if ($sessionObj->insert('session', $values, $rows)) {
$available_session['Session']->SessionId = $header['Session']->SessionId;
$available_session['Session']->SequenceNumber = $sequence;
$available_session['Session']->SecurityToken = $header['Session']->SecurityToken;
}
}
function that check availability of session in db :
public function getAvailableSession($session_data) {
$available_session = array();
foreach ($session_data as $key) {
if (!is_array($key)) {
$key = $session_data;
}
$date = date('Y-m-d h:i:s a', time());
$now = new DateTime($date);
$last_query_time = new DateTime($key['LastQueryDate']);
$dteDiff = $last_query_time->diff($now);
$difference = $dteDiff->format("%H:%I:%S");
//if response time out is smaller than session used last. then pick it.
if (RTO <= $difference) {
$available_session['Session']->SessionId = $key['SessionID'];
$available_session['Session']->SequenceNumber = $key['SequenceNo'];
$available_session['Session']->SecurityToken = $key['Security_token'];
// run update to update lastqueryDate as its default value set to current time stamp
$session_value = $key['SessionID'];
$rows['SequenceNo'] = $key['SequenceNo'];
$where[0] = "SessionID";
$where[2] = "'" . $session_value . "'";
$this->update('session', $rows, $where);
return $available_session;
}
}
return false;
}
As soon as i find a session sitting idle for more than 5 sec i am updating database.
Open transaction. Issue "select ... for update" query for getting session data. Commit transaction at end of script.