I've got SugarCrm plugin which is exporting data to external service. I'm using logic hooks for updated/deleted/new Contacts, but I've got problem with synchronizing already existing data. I have to extract all the data from the SugarCRM and there are two SugarBean methods I've tried to use: get_full_list() and get_list(). First one gives me the full Contact list, but I need to send it in batches 1000 Contacts in one Json max, the second method returns only first page of the Contacts (depends on config settings 10 - 1000max entries).
I'm using this method ATM:
// prepare contacts data from SugarBean
$bean = BeanFactory::getBean($module);
$contactResults = $bean->get_full_list();
Then foreach on $contactResults and save the data I want to the required format and send it as a Json via postrequest. I've tried to find the solution to split it into batches, but Im stuck :( Neither get_full_list or get_list seems to work for me.
Any suggestions? Maybe someone solved this issue already?
Thanks in advance!
It sounds to me like your problem is creating batches? If not please be more specific about what isn't working.
For splitting an array into batches, you may want to have a look at https://php.net/manual/en/function.array-chunk.php
Also get_list supports retrieving later pages. It is defined like this: function get_list($order_by = "", $where = "", $row_offset = 0, $limit=-1, $max=-1, $show_deleted = 0, $singleSelect=false, $select_fields = array()).
That means for the second page you could specify $row_offset = 1000, for the third page make it 2000, etc. So basically run a loop that calls get_list with $limit = 1000 and increases an initial $row_offset of 0 by 1000 after each iteration, until less than 1000 records or null is returned by the function.
Here are some general hints if you run into problems with processing those beans:
If the problem you're having is incomplete data, try loading each bean manually by using its ID. Some Sugar functions don't load all (special) fields by default.
If things seem to just fail for no reason, make sure to check your PHP log for errors. Maybe loading as many beans at once could possibly cause problems with your PHP's max_execution_time or memory_limit.
Related
Summary
This is a script (CakePHP 2.10.18 - LAMP dedicated server with PHP 5.3) that loads information from 2 MySQL tables, and then does some process of the data to output it to excel.
Table 1 has users, and Table 2 has info about those users (one record per user). The script has the goal of grabbing the record of a user from Table 1, grabbing its related info from Table 2, and put it in an excel row (using PHPExcel_IOFactory library for this).
The information extracted of those tables is of around 8000 records from each, the tables themselves have 100K and 300K total records respectively. All the fields in those tables are ints and small varchars with the exception of one field in the second table (datos_progreso seen in the code below), which is a text field and contains serialized data, but nothing big.
The issue is that if I run the script for the full 16000 records I get an Internal Server Error (without really any explanation in the logs), if I run the script for 1000 records it all works fine, so this seems to point out it's a resources issue.
I've tried (among other things that I will explain at the end) increasing the memory_limit from 128M to 8GB (yes you read that right), max_execution_time from 90 to 300 seconds, and max_input_vars from 1000 to 10000, and that isn't solving the problem.
My thoughts are that the amount of data isn't that huge to cause the resources to run out, but I've tried optimizing the script in several ways and can't get it to work. The only time I get it to work is by running it on a small portion of the records like I mention above.
I would like to know if there's something script-wise or php-configuration-wise I can do to fix this. I can't change the database tables with the information by the way.
Code
This is just the relevant bits of code that I think matter, the script is longer:
$this->Usuario->bindModel(
array('hasMany' => array(
'UsuarioProgreso' => array('className' => 'UsuarioProgreso', 'foreignKey' => 'id_usuario', 'conditions' => array('UsuarioProgreso.id_campania' => $id_campania)))
));
$usuarios = $this->Usuario->find('all', array(
'conditions'=>array('Usuario.id_campania'=>$id_campania, 'Usuario.fecha_registro >'=>'2020-05-28'),
'fields'=>array('Usuario.id_usuario', 'Usuario.login', 'Usuario.nombre', 'Usuario.apellido', 'Usuario.provincia', 'Usuario.telefono', 'Usuario.codigo_promocion'),
'order'=>array('Usuario.login ASC')
));
$usuario = null;
$progreso_usuario = null;
$datos_progreso = null;
$i = 2;
foreach ($usuarios as $usuario) {
if (isset($usuario['UsuarioProgreso']['datos_progreso'])) {
$datos_progreso = unserialize($progreso['UsuarioProgreso']['datos_progreso']);
$unit = 1;
$column = 'G';
while ($unit <= 60) {
if (isset($datos_progreso[$unit]['punt']))
$puntuacion = $datos_progreso[$unit]['punt'];
else
$puntuacion = ' ';
$objSheet->getCell($column.$i)->setValue($puntuacion);
$column++;
$unit++;
}
$nivel = 1;
$unidad_nivel = array(1 => 64, 2 => 68, 3 => 72, 4 => 76, 5 => 80, 6 => 84);
while ($nivel <= 6) {
$unidad = $unidad_nivel[$nivel];
if (isset($datos_progreso[$unidad]['punt']))
$puntuacion = $datos_progreso[$unidad]['punt'];
else
$puntuacion = ' ';
$objSheet->getCell($column.$i)->setValue($puntuacion);
$column++;
$nivel++;
}
}
//Free the variables
$usuario = null;
$progreso_usuario = null;
$datos_progreso = null;
$i++;
}
What I have tried
I have tried not using bindModel, and instead just load the information of both tables separately. So loading all the info of users first, looping through it, and on each loop grab the info for that specific user from Table 2.
I have tried also something similar to the above, but instead of loading all the info at once for the users from Table 1, just load first all their IDs, and then loop through those IDs to grab the info from Table 1 and Table 2. I figured this way I would use less memory.
I have also tried not using CakePHP's find(), and instead use fetchAll() with "manual" queries, since after some research it seemed like it would be more efficient memory-wise (didn't seem to make a difference)
If there's any other info I can provide that can help understand better what's going on please let me know :)
EDIT:
Following the suggestions in the comments I've implemented this in a shell script and it works fine (takes a while but it completes without issue).
With that said, I would still like to make this work from a web interface. In order to figure out what's going on, and since the error_logs aren't really showing anything relevant, I've decided to do some performance testing myself.
After that testing, these are my findings:
It's not a memory issue since the script is using at most around 300 MB and I've given it a memory_limit of 8GB
The memory usage is very similar whether it's via web call or shell script
It's not a timeout issue since I've given the script 20 minutes limit and it crashes way before that
What other setting could be limiting this/running out that doesn't fail when it's a shell script?
The way I solved this was using a shell script by following the advice from the comments. I've understood that my originally intended approach was not the correct one, and while I have not been able to figure out what exactly was causing the error, it's clear that using a web script was the root of the problem.
I am trying to get a reply text message from Clickatell using their Rest API, when I call the parseReplyCallback function when their system posts to my page - it seems to be null or I am not sure how to get the variables it is returning. What I would like to do is have all of the variables returned insert into a SQL database so I can use it elsewhere.
I have tried quite a few things, using various styles of getting the variables such as $_POST, $results['text'], $results->text, and so forth each time I can't seem to get any information out of it. I can't just var_dump or anything because I can't see any backend or console so I am pretty much in the blind, hoping someone else is using this system and has it working fine.
require __DIR__.'/clickatell/src/Rest.php';
use clickatell\ClickatellException;
use clickatell\Rest;
$Rest = new Rest("j8VKw3sJTZuVfQGVC7jdhA");
// Incoming traffic callbacks (MO/Two Way callbacks)
$Rest->parseReplyCallback(function ($result) {
//mysqli_query($con,"INSERT INTO `SMSCHAT` (`text`) VALUES ('$result')");
$mesageId = mysqli_real_escape_string($con,$result['messageId']);
$text = mysqli_real_escape_string($con,$result['text']);
$replyMessageId = mysqli_real_escape_string($con,$result['replyMessageId']);
$to = mysqli_real_escape_string($con,$result['toNumber']);
$from = mysqli_real_escape_string($con,$result['fromNumber']);
$charset = mysqli_real_escape_string($con,$result['charset']);
$udh = mysqli_real_escape_string($con,$result['udh']);
$network = mysqli_real_escape_string($con,$result['network']);
$keyword = mysqli_real_escape_string($con,$result['keyword']);
$timestamp = mysqli_real_escape_string($con,$result['timestamp']);
//do mysqli_query
});
I'd like for it to break the result into individual variables (because I plan on doing other things such as an auto-reply, etc) and upload it to the SQL database scrubbed.
Either doesn't create the table entry or gives me a blank one altogether in that first test where I put the result in the text field.
From a Clickatell point of view, although we understand what you're asking - it's unfortunately outside the scope of support that we offer on our products.
If you would like more information on our REST API functionality, please feel free to find it here: https://www.clickatell.com/developers/api-documentation/rest-api-reply-callback/
If you don't succeed in setting up the callbacks, please feel free to log a support ticket here: https://www.clickatell.com/contact/contact-support/ and one of our team members will reach out and try to assist where possible.
I have a query in a CakePHP 3.0 table with a formatResults() method applied to it.
In order to carry out this calculation, data is required from a table in a different database (Currencies).
I am currently using this code to gather the data and pass it to the function:
$currencies = TableRegistry::get('Currencies');
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
$currencyValues = $currencyValues->first()->toArray()
$query->formatResults(function ($results) use ($currencyValues) {
return $results->map(function($row) use ($currencyValues) {
$row['converted_earnings'] = $row['earned'] / $currencyValues['cur'.$row['currency']];
$row['converted_advances'] = $row['advances'] / $currencyValues['cur'.$row['currency']];
return $row;
});
});
The problem is, this code seems to take a very long time to execute, even though it is only iterating through a few hundred rows of data.
Further investigation revealed that if I do not collect the data from the 'currencies' table and instead declare $currencyValues as an array with fixed numbers the code takes a full second less to execute.
Through commenting out parts of the code, I have isolated this to be the cause of the problem:
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
If I remove this part of the code then everything runs quickly, and as soon as I add it in (even if I do not use the data it returns and use the static array) the page takes far longer to load. It should be noted that this problem occurs whether or not I use the ->cache() method, and that the query itself reports that it takes 0-1ms in the sql dump.
My question is - why is this slowing down my execution so much and how can I stop it? It is pretty much a requirement that I use the data from this table for the operation, so I am looking for a way to speed it up.
I am trying to run a query in BigQuery/PHP (using google php SDK) that returns a large dataset (can be 100,000 - 10,000,000 rows).
$bigqueryService = new Google_BigqueryService($client);
$query = new Google_QueryRequest();
$query->setQuery(...);
$jobs = $bigqueryService->jobs;
$response = $jobs->query($project_id, $query);
//query is a syncronous function that returns a full dataset
The next step is to allow the user to download the result as a CSV file.
The code above will fail when the dataset becomes too large (memory limit).
What are my options to perform this operation with lower memory usage ?
(I figured an option is to save the results to another table with BigQuery and then start doing partial fetch with LIMIT and OFFSET but I figured a better solution might be available..)
Thanks for the help
You can export your data directly from Bigquery
https://developers.google.com/bigquery/exporting-data-from-bigquery
You can use PHP to run a API call that does the export (you dont need the BQ tool)
You need to set the jobs configuration.extract.destinationFormat see the reference
Just to elaborate on Pentium10's answer
You can export up to a 1GB file in json format.
Then you can read the file line by line which will minimize the memory used by your application and then you can use json_decode the information.
The suggestion to export is a good one, I just wanted to mention there is another way.
The query API you are calling (jobs.query()) does not return the full dataset; it just returns a page of data, which is the first 2 MB of the results. You can set the maxResults flag (described here) to limit this to a certain number of rows.
If you get back fewer rows than are in the table, you will get a pageToken field in the response. You can then fetch the remainder with the jobs.getQueryResults() API by providing the job ID (also in the query response) and the page token. This will continue to return new rows and a new page token until you get to the end of your table.
The example here shows code (in java in python) to run a query and fetch the results page by page.
There is also an option in the API to convert directly to CSV by specifying alt='csv' in the URL query string, but I'm not sure how to do this in PHP.
I am not sure do you still using the PHP but the answer is:
$options = [
'maxResults' => 1000,
'startIndex' => 0
];
$jobConfig = $bigQuery->query($query);
$queryResults = $bigQuery->runQuery($jobConfig, $options);
foreach ($queryResults as $row) {
// Handle rows
}
This code works, but I just hacked it together with my limited knowledge of PHP and I'm sure there's a more elegant and efficient way to go about it. If you'd be so kind as to point out how I can improve, that would be great!
So I have a CSV file, structured like so:
Code Class Value Status Date Created Date Redeemed
========================================================================
a51f3g45 gold 50 valid 2012-08-20
4f6a2984 silver 200 redeemed 2012-08-23 2012-08-27
gf3eb54b gold 150 valid 2012-08-30
etc...
The user fills out a form to change the Class, Value, and Status fields of a given line. I cobbled together the following code to replace the old values with the new ones:
$file = 'codes.csv';
$old_csv_string = file_get_contents($file);
preg_match('/('.$_POST['code'].',.*,.*,.*,.*,.*)\n/',$old_csv_string,$matches);
$old_row = $matches[1];
preg_match('/'.$_POST['code'].',(.*,.*,.*),.*,.*\n/',$old_csv_string,$matches_part);
$old_row_part = $matches_part[1];
$new_row_part = $_POST['class'].",".$_POST['value'].",".$_POST['status'];
$new_row = str_replace($old_row_part,$new_row_part,$old_row);
$new_csv_string = str_replace($old_row,$new_row,$old_csv_string);
file_put_contents($file,$new_csv_string);
So can I do better than 10 lines of code? Any advice would be greatly appreciated :)
Note: I tried using fgetcsv, but I couldn't figure out how to find the unique code within a 2D array, then replace its siblings.
Why are you doing this ?
I think you should store the data in a SQL table.
Each time user update data, do it in the table.
If you want the CSV to be downloadable at any moment. Use a .htaccess to redirect your.csv to csv_generator.php only if your.csv does not exist.
csv_generator.php will regenerate the whole csv if it does not exist, save it on hard drive for later use, and send it with correct mime/type in header (so it's transparent for user). User don't see he is requesting a php page.
Then you need to delete the csv on hard drive each time someone update the data (so it will be regenerated on next request)
I think this is the way to have an always ready to download csv online.
Do you know google doc does this ? Users can change data in a spreadsheet wich is available to download as a csv from a url (you need to publish this spreadsheet as a csv file).
Try using split like that for each line:
list($code, $class, $value, $status, $created, $redeemed) = split(",", $line, 6) ;
Thus you will have each field in separate variable.
Of course you need to take care of the first row in case you don't want to copy header.