I have a script that perform the following steps
A user logs in, adds an SMS message and specifies recipients. That information is added to "queued message table";
Some process sends the messages using an API and messages are moved to "sent message table";
A delivery report comes and messages are deleted from "sent message table" and a log entry referencing sent messages is added to "sent message log table".
when a large quantity of messages is queued up in the "queued message table", steps (2) and (3) take a long time,
Before message is pushed to the API, a random unique id is generated for every recipients for later referencing when retrieving report, that id is used to table ' sent message log table'.
Below is the sample script
<?php
class Message {
/*
* random unique id for mobile number
*/
protected $mobile_ids = array();
public function add_unique_id($id, $mobile)
{
$this->mobile_ids[] = array($id, $mobile);
}
public function get_unique_id()
{
return $this->mobile_ids;
}
// The method that generated the xml for API
public function makeXML($param,$multi_part=false)
{
$xmlString =
"<SMS>
<authentification>
<username>sss</username>
<password>sss</password>
</authentification>
<message>
<sender>sender</sender>";
if($multi_part == "longSMS") $xmlString .= "<type>longSMS</type>";
$xmlString .= "<text>{$param['text']}</text></message><recipients>";
// Array of mobile numbers came from $param
$phone_numbers = $param['numbers'];
// Loop through the array and generate <gsm messageId='0001'>mobile</gsm>
foreach($phone_numbers as $mobile) {
// Generate id for mobile
$msg_id = $this->make_random_int();
/**
* This is the slow part of the script,
* IDs are added to the array for logging into the database
* When message is sent, i looped through the id and created a log for this message
**/
$this->add_unique_id($msg_id, $mobile);
$xmlString .= "<gsm messageId=\"{$msg_id}\">{$mobile}</gsm>";
}
$xmlString .= "</recipients></SMS>";
return $xmlString;
}
/**
* This is the method that created the log
* Log the sms
* You will need to call $msgid = $this->update_db('the sms')
* return value of $msgid is last_insert_id
*/
public function log_sms($msgid) {
// Log the currently sent message
$userData = array();
$now = date('Y-m-d H:i:s');
foreach ($this->mobile_ids as $data) {
$userData[] = "('{$msgid}', '{$data[0]}', '{$data[1]}', 'QUEUED', '0000-00-00', '0000-00-00', '{$now}')";
}
$query = 'INSERT INTO sent_sms_log (txtId,msgID,mobile,status,sentdate_time,deliver_date_time,sysdate_time) VALUES' . implode(',', $userData);
$this->ci->db->query($query);
$this->mobile_ids = array(); // reset the array
}
// Make random int
protected function make_random_int() {
$this->ci->load->helper('string');
$int = random_string('numeric', 12);
return $int;
}
/**
* Update database after sms sent
* #return int
*/
public function update_db($msg, $owner, $qid=0) {
$data = array('qid'=> $qid, 'sms' => $msg, 'date_time' => date('Y-m-d H:i:s'), 'owner' => $owner);
$this->ci->db->insert('f_sent_sms', $data);
return $this->ci->db->insert_id();
}
}
I'm guessing it could likely be the api you're working with. I've had to work with apis for different services that were extremely slow. Maybe try profiling different parts of the code with the benchmark class:
http://codeigniter.com/user_guide/libraries/benchmark.html
That'll be a quick and easy way to find the slowest parts of the code.
I'm guessing this is running on some kind of a loop right now? Instead of inserting an unknown number of records one at a time, check out the insert_batch() method for Active Record in the User Guide http://codeigniter.com/user_guide/database/active_record.html You can use one database call to insert all of your records. Instead of the loop inserting the data into the database, all the loop would have to do is build an array of all of the data that will be inserted. After the loop is done, run insert_batch('f_sent_sms', $my_data) for the array you just built.
It would be a good idea to benchmark it all too (before and after) as #Matthew already said.
Related
I have a php class
<?php
class Students
{
public $sesscode;
public $db_data;
function __construct($sesscode)
{
$this->sesscode = $sesscode;
$this->db_data = dbfetch(null, "SELECT * FROM students WHERE code = ?", [$this->sesscode]); //this simply returns data from db
$this->db_data = (empty($this->db_data)) ? $this->createProfile() : $this->db_data[0];
}
public function createProfile()
{
dbquery(null, "INSERT INTO students (code) VALUES (?)", [$this->sesscode]);
return dbfetch(null, "SELECT * FROM students WHERE code = ?", [$this->sesscode])[0];
}
public function updateAccount($row, $key, $val)
{
$data = json_decode($this->db_data[$row], true);
if (!$data) {
$data = [
$key => $val,
];
} else {
$data[$key] = trim($val);
}
dbquery(null, "UPDATE students SET account = ? WHERE code = ?", [json_encode($data), $this->sesscode]);
}
public function getData($row)
{
return json_decode($this->db_data[$row], true);
}
}
?>
Usage:
$student = new Student($sesscode); //sesscode is just some string
if (!empty($_POST)){
$student->updateAccount("account", "course", $_POST['course']);
//see class code/updateAccount method above for parameters. This means update dB table row "account", decode json and set course to post value of course
$student->updateAccount("account", "email", $_POST['email']);
$student->updateAccount("account", "phone", $_POST['phone']);
$student->updateAccount("account", "institution", $_POST['institution']);
}
My problem is, only the last line executes (and no, it isn't overwriting the data)
Moving the line that sets phone to take the place of institution will set phone and ignore those above it.
The line
$data = json_decode($this->db_data[$row], true);
resets the value of $data every time your updateAccount account function runs. It resets it to the value you fetched from the database when you created the object. Therefore this doesn't take account of any changes you've made in previous calls to updateAccount. So in fact it does overwrite your data, but probably just quite not in the way you imagined.
You need to either:
a) keep $data in-memory as a (private) property of the class rather than resetting it each time updateAccount runs,
or
b) re-fetch the existing data from the database each time updateAccount runs,
or
c) Store your data in relational format (with fields in separate columns) instead of as JSON within a single column - meaning you could update individual fields without interfering with the others,
and/or
d) provide a version of updateAccount where multiple values can be updated simultaneously - running separate UPDATE queries for each field is very inefficient.
I have a loading script to insert JSON files into a database, this script is executed almost 50 times using AJAX and a loading indicator. I want to use the below structure of insert to get this done as fast as possible to avoid hunderds of single insert statement executed
INSERT INTO TABLE (COLUMN) VALUES ('value1'),('value2'),('value3')
To generate the SQL in laravel as the example here above, I use this code:
foreach ($album_data['data'] as $album) {
if (isset($album['name'])) {
$array['id'] = $album['id'];
$array['name'] = $album['name'];
$array['type'] = $album['type'];
$arr_albums[] = $array;
unset($array);
}
}
DB::table('ALBUM')->insert($arr_albums);
This works most of the time, but sometimes an error occurs regarding duplicate keys because this script is fired a few times at once for other JSON inputs where the same albums appear, even if I first check for already existing ID's.
This is the proces currently:
get all ID's from the JSON file
use these ID's in a SELECT to check wich are already in the database
create the INSERT statement for those ID's who are not in the database
The error occurs when 2 or more AJAX requests with same album ID's in the JSON files are checking (number 2 of the proces) at the same time, and running (number 3 of the proces) insert at the same time.
I try to fix this by adding "ON DUPLICATE KEY UPDATE..." at the end of the generated SQL, so I assumed to do it like this:
$query = DB::table('ALBUM')->insert($arr_albums)->toSql();
$query .= " ON DUPLICATE...";
what causes this error:
Call to a member function toSql() on boolean
I guess the "toSql()" function is not possible on "insert()" ?
I think the easiest way is to use the table object's Grammar:
$table = \DB::table('foo');
$sql = $table->getGrammar()->compileInsert($table, $mapping);
$updateSQL = $table->getGrammar()->compileUpdate($table, $mapping);
\DB::query($sql . ' ON DUPLICATE KEY ' . substr($updateSQL, strpos($updateSQL, ' set ')));
/Illuminate/Database/Query/Grammars/Grammar has functions:
/**
* Compile an insert statement into SQL.
*
* #param \Illuminate\Database\Query\Builder $query
* #param array $values
* #return string
*/
public function compileInsert(Builder $query, array $values)
/**
* Compile an update statement into SQL.
*
* #param \Illuminate\Database\Query\Builder $query
* #param array $values
* #return string
*/
public function compileUpdate(Builder $query, $values)
I am making a transaction table with an auto increment field of BIGINT(20).
When a new transaction is added, the insert Id is retrieved and formatted to be more readable:
public function add_transaction($paymethod, $cursus_id)
{
$this->load->model('Config_model');
$btw = $this->Config_model->get('transactions.btw');
$query = " INSERT INTO transacties(userid, paymethod, amount, btw_pc)
VALUES((SELECT userid FROM users WHERE lcase(username)=lcase('{$this->session->userdata('username')}')),
'{$paymethod}',
(SELECT prijs FROM p_cursus_uitvoering WHERE uitvoering_id = {$cursus_id}),
{$btw});";
$this->db->query($query);
$insertId = $this->db->insert_id();
$newCode = date('Ymd') . str_pad($insertId, 8, '0', STR_PAD_LEFT);
$this->db->where('transact_id', $insertId);
$this->db->update('transacties', ['transact_id' => $newCode]);
return $newCode;
}
The result is that the ID get updated from eg: 5 to 2015041800000005.
This is working perfectly, but as you can see, the newCode is returned in the function and used by another function where it's reinserted in another table.
Here is where the problem arises, the ID turns into: 201504192058506757.
Even when I echo the newCode, it still prints 201504192058506757.. even though it is inserted correctly once, but incorrectly the second time!
EDIT:
Here is the code snippet in which the function is called:
public function workshop(){
$this->load->model('Inschrijven_model');
$paymethod = $this->input->post("paymethod");
//eigenlijk uitvoering_id.....
$cursus_id = $this->input->post("cursus_id");
if($paymethod == null || $cursus_id == null){
redirect('cursus');
}
if($this->Inschrijven_model->cursus_has_room($cursus_id)){
if(!$this->Inschrijven_model->cursus_ingeschreven($cursus_id)){
$this->load->model('Cc_payment_model');
$this->load->model('Cursus_model');
$amount = $this->Cursus_model->get_price($cursus_id);
$orderId = $this->Cc_payment_model->add_transaction($paymethod, $cursus_id);
$this->Inschrijven_model->cursus_inschrijven($cursus_id,$orderId);
$function = explode('_',$paymethod);
$this->{$function[0]}($function[1], $amount, $orderId);
}else{
echo "Al ingeschreven";
}
}else{
echo "geen ruimte";
}
}
And here is the code snippet in which the return $newCode is being reinserted:
public function cursus_inschrijven($cursus_id,$transaction_id){
$query = " INSERT INTO p_cursus_in(uitvoering_id, userid, transact_id)
VALUES({$cursus_id},(SELECT userid FROM users WHERE lcase(username)=lcase('{$this->session->userdata('username')}')),{$transaction_id})";
$this->db->query($query);
}
It is reinserted so I can make a connection between someone's registration into a class and their payment for that class.
You have to convert the BIGINT to string before you get it into PHP
Here's how you can convert it with a SQL statement:
https://dev.mysql.com/doc/refman/5.7/en/cast-functions.html#function_convert
I don't know what framework you're using but you have to do a conversion if you're using BIGINT whose length is not supported by PHP directly. It seems that the framework will read the record after updated to check if the operation's done successfully, and because of the process of reading without the conversion, you got a wrong number.
You can also use PHP GMP module to handle BIGINT:
https://php.net/manual/en/book.gmp.php
after searching for a long time got this great article its really very nice
but i am facing a bit problem here in my stuff as u have used direct mysql query in api i have used stored procedure in here and every time i have to compare two XML before and after even for a single short and sweet query so is there any alternative for this process but which is this secure
please chk this out u will get i more clearly
database testing in php using phpunit,simpletest on api haveing stored procedure
or how shall i compare to xml files before and after api function call(the function contains the stored procedure)
means i am able to get the before state with mysql-dump but the after but not getting the instant after xml state
sorry for the English but tried my best
thanks for the help friend
have to write an unit test test for the api function
public function delete($userId)
{
// this function calls a stored procedure
$sql = "CALL Delete_User_Details(:userId)";
try {
$db = parent::getConnection();
$stmt = $db->prepare($sql);
$stmt->bindParam("userId", $userId);
$stmt->execute();
$id = $stmt->fetchObject();
if ($id == null) {
$delete_response->createJSONArray("DATABASE_ERROR",0);
} else {
$delete_response->createJSONArray("SUCCESS",1);
}
} catch (PDOException $e) {
$delete_response->createJSONArray("DATABASE_ERROR",0);
}
return $delete_response->toJSON();
}
i have writen this unit test for it now want to write an dbunit for it
public function testDeleteUser()
{
$decodedResponse = $response->json();
$this->assertEquals($response->getStatusCode(), 200);
$this->assertEquals($decodedResponse['status']['StatusMSG'], 'SUCCESS');
$this->assertEquals($decodedResponse['status']['Code'], '1');
}
help guyss
u can just simply test it before by calling the query like
$sql = "select * from user";
and compare it with BeforeDeleteUser.xml
And the Call Ur stored procedure
$sql = "CALL Delete_User_Details(:userId)";
And for the after case just repeat the before one again
$sql = "select * from user";
and compare it with AfterDeleteUser.xml
see the logic is very simple if u have 5 Users in BeforeDeleteUser.xml and it results true and after the call of CALL Delete_User_Details(:userId) stored procedure , the AfterDeleteUser.xml should contain only 4 user (or maybe idDelete field to 0 that depends on ur implementation)
I'm building an iPhone push server, and trying to get the push to work. I have a message.php file which put new message in the database, and then add the message to a push_queue table in the database.
To send the push, I manually have to go to the browser and call the push file (../push/push.php) which will send out the push.
Is there any way I can call the push.php file from the message.php file automatically?
I tried require_one, include, exec and file_get_contents without any luck.
It works if I use:
header('Location: ../push/push.php');
However, the push.php file takes a couple of seconds to execute and finish, so there's a delay for the user when trying to send a message.
I guess I could use a cron job to call the push.php file, but I'd rather not.
Here is the core function in push.php (based on http://www.raywenderlich.com/3525/apple-push-notification-services-tutorial-part-2):
function start()
{
//writeToLog('Connecting to ' . $this->server);
if (!$this->connectToAPNS())
exit;
while (true)
{
// Do at most 20 messages at a time. Note: we send each message in
// a separate packet to APNS. It would be more efficient if we
// combined several messages into one packet, but this script isn't
// smart enough to do that. ;-)
$stmt = $this->pdo->prepare('SELECT * FROM push_queue WHERE time_sent IS NULL LIMIT 20');
$stmt->execute();
$messages = $stmt->fetchAll(PDO::FETCH_OBJ);
$deletedIds = array();
foreach ($messages as $message)
{
if ($this->sendNotification($message->message_id, $message->device_token, $message->payload))
{
//$stmt = $this->pdo->prepare('UPDATE push_queue SET time_sent = NOW() WHERE message_id = ?');
//$stmt->execute(array($message->message_id));
$deletedIds[] = $message->message_id;
//$stmt = $this->pdo->prepare('DELETE FROM push_queue WHERE message_id = ?');
//$stmt->execute(array($message->message_id));
}
else // failed to deliver
{
$this->reconnectToAPNS();
}
}
//Delete the chunk of messages.
$this->pdo->query('DELETE FROM push_queue WHERE message_id IN ('.implode(',', $deletedIds).')');
unset($messages);
}
}
Create a function or class that does everything that your push.php does and call it when a new message is received or when the iPhone app queries for new messages. In this case you will not need to call other PHP in message.php.
This is a concept of MVC i.e. having your business logic separated from your controllers. In this case pushing is a business logic and message.php and push.php are your controllers.