PDO 'too many connections' when I select all rows from DB - php

I am trying to write a function that verifies a bank transaction amount against all purchases linked to that transaction. It is possible to link a purchase to more than one transaction and for more than one purchase to link to the same transaction.
The function works by totalling the transactions in question, then running a DB query to select all purchases, then checking each purchase to see if it references any of the transactions in question. It then totals the amount of all purchases linked to any of the transactions and reconciles it by checking if the two totals add up.
My problem is that I always get a 'too many connections' error when I get all the purchases from the database. It's fine if I limit the search, but I need all of them in order for the function to work.
I either need to fix the connection overload issue somehow write the query to only search for purchases containing those transactions - but I'm not sure how to do that.
The transactions are comma separated in a column of the purchase entry.
Interestingly, if the list of purchases is called via AJAX it works fine. But if the page (including other prior connections) is loaded statically - it doesn't work. I'm assuming that because the AJAX is loading one thing, there are no prior connections in that instance.
It is quite a complicated system so this may not be very helpful on its own:
public static function verify($ids, $return = NULL) {
// Transaction total
$transactions = explode(",",$ids);
$transTotal = 0;
foreach($transactions as $transaction) {
$transTotal = $transTotal + self::get($transaction,"amount");
}
// Expense/item total
$accounts = AccItem::getAll("all");
$itemTotal = 0;
foreach($accounts as $item) {
$translink = explode(",",$item->transaction_ids);
if(array_intersect($transactions, $translink)) {
$itemTotal = $itemTotal + AccItem::calculate($item->id,"total") + AccItem::calculate($item->id,"tax");
}
}
unset($accounts);
if($transTotal == $itemTotal or $transTotal + $itemTotal == 0) {
if($return) return 'check';
if(!$return) echo '<abbr title="Transaction verified."><i class="fa fa-check-circle"></i></abbr>';
} else {
if(!$return) echo '<abbr title="Transaction amount mismatch!"><i class="fa fa-exclamation-circle"></i></abbr>';
}
}
And here is the getAll function:
public static function getAll($chart_id, $date_from = 0, $date_to = 9999999999999) {
$db = new Data;
if($chart_id == "all") {
$sql = $db->query("SELECT * FROM mc_account_items WHERE date_incurred >= :date_from AND date_incurred <= :date_to ORDER BY date_incurred DESC");
} else {
$sql = $db->query("SELECT * FROM mc_account_items WHERE chart_id = :chart_id AND date_incurred >= :date_from AND date_incurred <= :date_to ORDER BY date_incurred DESC");
$sql->bindParam(":chart_id", $chart_id);
}
$sql->bindParam(":date_from", $date_from);
$sql->bindParam(":date_to", $date_to);
$sql->setFetchMode(PDO::FETCH_CLASS | PDO::FETCH_PROPS_LATE,'AccItem');
$sql->execute();
return $sql->fetchAll();
//unset($db);
}

So I read into things a bit more, turns out I wanted a persistent connection as it was the same connection every time. Instead of opening a new one it will just use the cached connection.
PDO::ATTR_PERSISTENT => true

Related

Recursive call for cron job

I have a three PHP script, from which two are running as a separate cron job each day in server.
First script(get_products.php), make curl call to external server to get all products data and stored that data in database. Note that there are around 10000 products and increasing each day.
In Second script(find_related.php), selects products from database stored by first script, perform some operations and store operational data in another database. Each product have 10 rows so in this database there is around 100000 rows. This script is running as cron. Sometimes the script is not executed fully and that's why, the actual and expecting results are not stored in database. I included this line of code in script: ini_set('max_execution_time', '3600');
But it not works.
Here is the process done in this script:
Normally task is to find 10 related products based on tags. I have around 10300 products stored in my DB. Each time query take one product and their tags and try to randomly find one product tagged with same tag as main product and store the related product data into another DB for third script. Only one product per tag is allowed. If it will not find total of 10 related products then randomly gets products from another DB named bestseller_products.
Here is my code:
$get_all_products = mysql_query('SELECT * FROM store_products');
while($get_products_sql_res = mysql_fetch_array($get_all_products)){
$related_products = array();
$tags = explode(",",$get_products_sql_res['product_tags']);
$product_id = $get_products_sql_res['product_id'];
$product_handle = $get_products_sql_res['product_handle'];
$get_products_sql = mysql_query('SELECT * FROM related_products WHERE product_handle="'.$product_handle.'"');
if (mysql_num_rows($get_products_sql)==0)
{
$count = 0;
foreach($tags as $t){
$get_related_products_sql = mysql_query("SELECT product_handle, product_title, product_image FROM store_products WHERE product_tags like '%".$t."%' AND product_id != '".$product_id."' ORDER BY RAND()");
if(!$get_related_products_sql){
continue;
}
while($get_related_products = mysql_fetch_array($get_related_products_sql) ){
$related_product_title = mysql_real_escape_string($get_related_products['product_title']);
$found = false;
foreach($related_products as $r){
if($r['handle'] == $get_related_products['product_handle']){
$found = true;
break;
}
}
if($found == false){
$related_products[$count]['handle'] = $get_related_products['product_handle'];
mysql_query("INSERT INTO related_products (product_handle, product_id, related_product_title, related_product_image, related_product_handle) VALUES ('$product_handle','$product_id','$related_product_title', '$get_related_products[2]', '$get_related_products[0]')");
$count = $count + 1;
break;
}
}
}
if($count < 10){
$bestseller_products = mysql_query("SELECT product_handle, product_title, product_image FROM bestseller_products WHERE product_id != '".$product_id."' ORDER BY RAND() LIMIT 10");
while($bestseller_products_sql_res = mysql_fetch_array($bestseller_products)){
if($count < 10){
$found = false;
$related_product_title = mysql_real_escape_string($bestseller_products_sql_res['product_title']);
$related_product_handle = $bestseller_products_sql_res['product_handle'];
foreach($related_products as $r){
if($r['handle'] == $related_product_handle){
$found = true;
break;
}
}
if($found == false){
$related_product_image = $bestseller_products_sql_res['product_image'];
mysql_query("INSERT INTO related_products (product_handle, product_id, related_product_title, related_product_image, related_product_handle) VALUES ('$product_handle','$product_id','$related_product_title', '$related_product_image', '$related_product_handle')");
$count = $count + 1;
}
}
}
}
}
}
Third script(create_metafields.php), created metafields in external server using data created by second script. And same problem arises as in second script.
So i want to execute the second script into parts. I mean, not to process all 10000 products in one call but want to run unto parts(1-500,501-1000,1001-1500,..) like it. But dont want to create separate cron jobs. Please suggest if someone has solution. I really need to figure it out.
Thanks in advance!

Pagination with token in BigQuery + PHP client not working

I'm trying to implement a simple pagination mechanism with BigQuery.
this is my query function:
function query($sql, $max_results = null, $page_token = null) {
$request = new Google_Service_Bigquery_QueryRequest();
$request->setQuery($sql);
$response = $this->service->jobs->query(PROJECT_ID, $request);
$job_id = $response->getJobReference()->getJobId();
$optParams = ($max_results) ? array(
'pageToken' => $page_token,
'maxResults' => $max_results,
) : array();
$response = $this->service->jobs->getQueryResults(PROJECT_ID, $job_id, $optParams);
if (!$response->getJobComplete()) {
return null;
}
$rowsJson = $this->rowsJson($response->getRows());
if ($max_results) {
return array(
"rows" => $rowsJson,
"token" => $response->getPageToken()
);
}
return $rowsJson;
}
query("select url, CEIL(AVG(total)) as avg, count(id) as count from $table_id " .
"where created > $date_start and created < $date_end group by url order by $order_by desc",
10, $page_token
);
The first query seems like it's working. it returns only 10 results and a token (there are definitely more than 10 results). But when I call the function later with the token I got, the results are empty... I have no idea why. I see no error message in the logs or anything...
In order for BigQuery "pagination" to work you shold follow below steps (simplified but enough for this answer):
Execute query job
Retrieve jobid
Execute getQueryResults with that jobid and retrieve result and page_token
If page_token null - you done, otherwise continue with next step
Execute getQueryResults (still with jobid from #2) and retrieve result and page_token
Jump to step #4
Now, you most likely will see that your code doesn't follow this and each time executes query as new job - which totally breaks condition highlighted in #5 (still with jobid from #2).
The reason why first time it returns result is that first call actually meet above steps, but in consequitive calls you actually passing page_token and force execution of NEW job with some arbitrary (from that new job prospective) page_token
And also, this explains why your current code works with startIndex
Try to rewrite your code to follow above steps and it should work
Also,
P.S. page_token is alive and should be reusable for as long as underlying temp table (that holds your result) is alive - which is approximately 24 hours
Solved it by using startIndex instead of tokens.
Maybe tokens are short-lived and don't survive the 4-5 seconds delay, I'm not really sure.
Using a page token to get paged results are recommended because page token provides a view of the table at the first time it is retrieved. By just using startIndex, you may see different results as the table may change.
Response size is what may cause you get less than maxResults. We are limited by Apiary to return results of certain size.
https://cloud.google.com/bigquery/docs/data#paging
And we also have a max field counts to be less than 350,000 limit per call.
You should try to get the next results by using the page token it passed back to you.
function get_quote($page = 1){
global $database_connection;
$start = 0;
$limit = 5;
$start = ($page - 1) * $limit;
//select from database with limit $start, $limit
//display results
$sql_get_quote = "SELECT * FROM quote_table ORDER BY date_added ASC LIMIT $start, $limit";
$query_get_quote = mysqli_query($database_connection, $sql_get_quote);
while($fetch_quote = mysqli_fetch_array($query_get_quote)){
echo "<br>
<div class='w3-container w3-pale-green w3-bottombar w3-border-green w3-border'>
<p>".$fetch_quote['quote']."</p><hr><p class='w3-left-align'>".$fetch_quote['quote_by']."</p>
</div>";
}
//do another query, get the num_rows
$quote_rows= mysqli_num_rows (mysqli_query($database_connection ,"SELECT * FROM quote_table"));
$total = ceil($quote_rows / $limit);
if (isset($page)){
echo "<div class='w3-container'>";
echo "<ul class='pager'>";
if($page > 1) {
echo "<a href='?page=".($page - 1)."' style='float: left;' class=' w3-sand w3-btn w3-margin w3-round'>Previous</a>";
}
for($i = 1; $i <= $total; $i++) {
if($i == $page) { echo "<li class='active current'>".$i."</li>"; }
else { echo "<li><a href='?page=".$i."'>".$i."</a></li>"; }
}
if($page != $total) {
echo "<a href='?page=".($page + 1)."' class='w3-btn w3-margin w3-sand w3-round'>Next</a>";
}
echo "</ul></div>";
}
}
This works for me... Hope you can build around this

Quickbooks - Select all payments for an invoice

I am trying to query for all payments assigned to an invoice using the consolibyte quickbooks php toolkit. I was hoping I could pull these back using a query but so far the only way I can see to do it is by grabbing all payments and then looping through and checking what is within the LinkedTxn array and checking if it has a type of 'Invoice' and that the 'TxnId' value matches my stored invoice id. What I have so far is this which gets the payments but is obviously not an option going forward as there may be tens of thousands of payments in the system:
public function getAllPaymentsForInvoice(Invoice $invoice)
{
$query = "SELECT * FROM Payment";
$payments = $this->qbPaymentService->query($this->context, $this->realm, $query);
$lines = [];
foreach ($payments as $payment) {
$num_lines = $payment->countLine();
for ($i = 0; $i < $num_lines; $i++) {
$line = $payment->getLine($i);
$txnId = $this->parseResponse($line->getLinkedTxn()->getTxnId());// convert {-1} to 1
$txnType = $line->getLinkedTxn()->getTxnType();
if ($txnType == 'Invoice' && $txnId == $invoice->qb_ref) {
$lines[] = $line;
}
}
}
return $lines;
}
Can anyone push me in the direction of a better way?
There's no easy, one-liner way to do this using the QuickBooks Online API. This just isn't something that QuickBooks Online itself supports.
There are two separate approaches you could take to optimize what you're doing.
The payments applied should always be for the same customer as the invoices, so you can change your query to SELECT * FROM Payment WHERE CustomerRef = 'xyz' where xyz is the CustomerRef from the invoice. Then, use your existing code to check exactly which invoices payments are applied to.
Use the CDC functionality to keep a cached copy of all payments in your own database, and query your own database for the information instead (a normal SQL database is considerably more flexible and query-able than what QuickBooks Online offers).
The CDC functionality is specifically designed to allow you to keep an accurate, cached copy of the QuickBooks data in your own application, specifically to address situations like you're running into. You pass it a timestamp, and it gives you everything that has changed since that timestamp. By remembering the date/time you last ran the ->cdc(...) method, you can continually get lightweight updates to any objects that have changed since you last queried them.
<?php
$CDCService = new QuickBooks_IPP_Service_ChangeDataCapture();
// What types of objects do you want to get?
$objects = array(
'Payment',
'Invoice',
);
// The date they should have been updated after
$timestamp = QuickBooks_Utilities::datetime($datetime_you_last_called_cdc_method);
$cdc = $CDCService->cdc($Context, $realm,
$objects,
$timestamp);
// You would cache all the stuff you get back here
print_r($cdc);
Docs links:
https://github.com/consolibyte/quickbooks-php/blob/master/docs/partner_platform/example_app_ipp_v3/example_cdc.php
https://developer.intuit.com/docs/0100_accounting/0300_developer_guides/change_data_capture
The question is quite old, anyway, in case someone else is looking for answer:
$InvoiceService = new QuickBooks_IPP_Service_Invoice();
// $invoices = $InvoiceService->query($Context, $realm, "SELECT * FROM Invoice STARTPOSITION 1 MAXRESULTS 10");
// If, we have QB Invoice # i.e.DocNumber
$invoice_no = 1001;
$invoices = $InvoiceService->query($Context, $realm, "SELECT * FROM Invoice WHERE DocNumber = '$invoice_no' ");
foreach ($invoices as $Invoice)
{
$txnId = 0;
if (is_object($Invoice->getLinkedTxn()) && $Invoice->getLinkedTxn()->getTxnType() == 'Payment') {
$txnId = $Invoice->getLinkedTxn()->getTxnId();
}
print('Invoice # '.$Invoice->getDocNumber().' has a total of $'.$Invoice->getTotalAmt().' and Linked Payment ID: ' . $txnId . "\n");
}

QuickBooks php DevKit database query only returning 100 rows

I am trying to use the DevKit to go through all existing customers in my QuickBooks Online account and update the display name for each one of them to prepend their company name with the company Id I have assigned them. Currently, there are about 1800 customers in the account.
Using the code below, I can successfully process exactly 100 of the customers each time I run the script, but it stops at this point. Is there some time or number limit to the rows return by the Service Customer query? If so, is there a way to increase it? Or is the problem something else entirely?
Quickbooks_update:
class Quickbooks_Update extends CI_Controller
{
function quickbooks_add(){
require_once 'application/QuickBooks/config.php';
$this->load->model('companyAccounts');
$success = array();
$failed = array();
$duplicate = array();
$not_exist = array();
$CustomerService = new QuickBooks_IPP_Service_Customer();
$customers = $CustomerService->query($Context, $realm, "SELECT * FROM Customer ");
foreach($customers as $customer) {
$name = $customer->getCompanyName();
$q = $this->db->select('companyAccountId');
$q = $this->db->from('companyAccounts');
$q = $this->db->where('companyName', $name);
$q = $this->db->get();
$results = $q->result();
foreach($results as $company){
$companyId = $company->companyAccountId;
}
if(sizeof($results) == 1){
$customer->setDisplayName($companyId . '-' . $name);
$resp = $CustomerService->update($Context, $realm, $customer->getId(), $customer);
if(!$resp){
array_push($failed, $name);
}else{
array_push($success, $name);
}
}
else if(sizeof($results) == 0){
array_push($not_exist, $name);
}
else{
array_push($duplicate, $name);
}
}
$data['success'] = $success;
$data['failed'] = $failed;
$data['duplicate'] = $duplicate;
$data['not_exist'] = $not_exist;
$this->load->view('quickbooks', $data);
}
}
Thank you!
QuickBooks Online by default only returns 100 rows at a time.
If you refer to the documentation:
https://developer.intuit.com/docs/0100_accounting/0300_developer_guides/querying_data
You'll find a bunch of examples of how to use STARTPOSITION and MAXRESULTS to control how many records you get back:
SELECT * FROM Invoice WHERE Status = 'Synchronized'STARTPOSITION 1 MAXRESULTS 10
SELECT * FROM Invoice WHERE Status = 'Synchronized'STARTPOSITION 11 MAXRESULTS 10
To quote the documentation:
To page through the results, specify STARTPOSITION (position of the entity in the query results) and MAXRESULTS (maximum number of entities in the result).
And:
The maximum number of entities that can be returned in a response is 1000. If the result size is not specified, the default number is 100. If a query returns many entities, fetch the entities in chunks, as described in Pagination. To determine the number of entities that a particular query returns, probe by using the COUNT keyword in the query. See Count for details.

Array and for each loop issues

Code is below if I run one value in the array the results are correct if I run more than one value the results are of the price is incorrect its like it has messed around with the values somewhere ?? help appreciated
$dido=array('42204131','22204131');
foreach($dido as $did):
$query = "select * from dispatch,link where lid=dlid and did=$did";
$result = mysql_query($query) or die(mysql_error());
while($row = mysql_fetch_array($result)){
$vanc1=$row['vanc1'];
$vanc2=$row['vanc2'];
$vanc3=$row['vanc3'];
$vanc4=$row['vanc4'];
$vanc5=$row['vanc5'];
$anc1=$row['anc1'];
$anc2=$row['anc2'];
$anc3=$row['anc3'];
$anc4=$row['anc4'];
$anc5=$row['anc5'];
// price anc1
$querypanc1 = "select pprice from products where pid=$anc1";
$resultpanc1 = mysql_query($querypanc1);
while($row = mysql_fetch_array($resultpanc1))
{
$priceanc1=$row[pprice];
$tpriceanc1=$vanc1*$priceanc1;
}
// price anc2
$querypanc2 = "select pprice from products where pid=$anc2";
$resultpanc2 = mysql_query($querypanc2);
while($row = mysql_fetch_array($resultpanc2))
{
$priceanc2=$row[pprice];
$tpriceanc2=$vanc2*$priceanc2;
}
// price anc3
$querypanc3 = "select pprice from products where pid=$anc3";
$resultpanc3 = mysql_query($querypanc3);
while($row = mysql_fetch_array($resultpanc3))
{
$priceanc3=$row[pprice];
$tpriceanc3=$vanc3*$priceanc3;
}
// price anc4
$querypanc4 = "select pprice from products where pid=$anc4";
$resultpanc4 = mysql_query($querypanc4);
while($row = mysql_fetch_array($resultpanc4))
{
$priceanc4=$row[pprice];
$tpriceanc4=$vanc4*$priceanc4;
}
// price anc5
$querypanc5 = "select pprice from products where pid=$anc5";
$resultpanc5 = mysql_query($querypanc5);
while($row = mysql_fetch_array($resultpanc5))
{
$priceanc5=$row[pprice];
$tpriceanc5=$vanc5*$priceanc5;
}
$gtprice=$tpriceanc1+$tpriceanc2+$tpriceanc3+$tpriceanc4+$tpriceanc5;
$qrygt="UPDATE dispatch SET gtprice=$gtprice WHERE did=$did";
$resultgt=#mysql_query($qrygt);
}
endforeach;
1) The only possible issue I could spot in your code, is that when some of your select pprice from products where pid ... queries do not return any data, you retain value of $tpriceancX from previous iteration.
2) Also (out of topic) you can replace your 5 blocks of repeated code with for loop.
$gtprice = 0;
for ($i = 1; $i <= 5; $i++)
{
$querypanc = "select pprice from products where pid=".$row["anc$i"];
$resultpanc = mysql_query($querypanc);
while($pancrow = mysql_fetch_array($resultpanc))
{
$priceanc=$pancrow[pprice];
$tpriceanc=$row["vanc$i"]*$priceanc;
$gtprice += $tpriceanc;
}
}
Your first and biggest problem is the copy-pasta nature of your code. Let's try and break down what you're doing:
Setting up a list of ids
Running a query on those ids
Putting the results into an array
Running a separate query on each of those results
You are also using some very janky syntax. (ie foreach($foo as $bar):).
Break these things down into methods. What is a method? It takes an input and transforms it into an output.
//returns an array of price information
public function getPrices($idArray) { //note the good method and parameter names!
//do stuff
}
Now that we know what we are doing, we can start to fill in the implementation details:
public function getPrices($idArray) {
foreach($idArray as $id) {
//somehow get the gross-scale information
//then put it in a data object
//then call a function to get specific information
}
}
What should that sub-method do? Lets look at your current code snippet:
// price anc1
$querypanc1 = "select pprice from products where pid=$anc1";//sets up sql query
$resultpanc1 = mysql_query($querypanc1); //runs the query
while($row = mysql_fetch_array($resultpanc1)) { //for each result
$priceanc1=$row[pprice]; //gets the price
$tpriceanc1=$vanc1*$priceanc1; //calculates some other price
}
Those last two lines really suggest an object but maybe that's too heavyweight for your purpose. The first two lines are boiler plate you repeat endlessly. Lets write a function!
public function getPrices($name, $pid, $multiplier) {
$sqlQuery = "SELECT pprice FROM products WHERE pid=$pid";
$result = mysql_query($sqlQuery);
$prices = array();
while ($row = mysql_fetch_array($result) {
$key = "price".$name;//$key will be something like 'priceanc1'
$prices[$key] = $row[pprice];
$tkey = "tprice".$name;
$prices[$tkey] = $prices[$key] * $multiplier;
}
}
Now, this function is a bit unclean because it tries to do two things at once (queries the database and then massages the data into a usable array) but I wanted it to resemble what you were doing. With this function written we can go back to our higher level function an call it:
public function getPrices($idArray) {
foreach($idArray as $id) {
$sqlQuery = "SELECT * from dispatch, link WHERE lid=dlid and did=$id";
$prices = array();
while ($row = mysql_fetch_array($result) {
for ($idx = 1; $idx <= 5; $idx++) {
$name = "anc".$idx;
$pid = $row[$name];
$multiplier = $row["vanc".$idx];
$priceArray = getPrices($name, $pid, $multiplier);
$prices = array_merge($prices, $priceArray);
}
}
}
//put a var_dump here to check to see if you're getting good results!
return $prices;//Should be the aggregated prices you've gotten from the db
}
Now, that is what you're attempting to do, but I admit I don't understand how your database is set up or what your variables actually mean. Pressing on! We also note that unnecessary massaging of data falls away.
You can call this like so:
$ids = array();
$ids[] = 42204131;
$ids[] = 22204131;
$prices = getPrices($ids);
var_dump($prices);//shows the result of your work
Now that you have the prices, you can pass them to another function to run the update:
updatePrices($prices);
I'll let you write that part on your own. But remember; break down what you're doing and have repeated elements be handled by the same function. The real lesson to learn here is that programming is really communicating: your code doesn't communicate anything because there is so much repeated noise. Use good variable names. Tighten what you're doing down to functions with single tasks. This way anyone reading your code (including you!) will know what you're trying to do and where you've gone wrong.

Categories