PHP Batch upload array data - php

Question

I am working with a client API (master API) that does not have a bulk feature.
I have taken data from 2 different API's (client API's) and merged it into one JSON file that is properly formatted. Checked in online JSON Validator.
The JSON File is 1100 records of merged customer data. Taking one record at a time, I have built a function that submits the data successfully to the master API.
I have now built a PHP script that loops through the JSON File and takes the row data (each client record) and submits it to the master API successfully. After about 90 rows, the PHP script times out.
I have set the following code on the page
#ini_set('zlib.output_compression', 0);
#ini_set('implicit_flush', 1);
set_time_limit(600);
#ob_end_clean();
And am buffering each update to return a JSON status code returned from the master API.
What should I be doing to get the PHP to not time out after about 100 records and keep updating the buffer response on the page.?
Thanks in advance.
Jason

Answer 1

I've taken a few of different approached to this kind of problem in the past. The only quick fix for this is if you can change the php.ini settings on the server to increase the timeout enough to allow your batch to complete. This is not a great solution, but it is a solution.
The next option (in ascending order of effort) is to set up a loop between the browser and your server where your browser makes a request, the server sends a portion of the records, then returns to the browser with a cursor indicating where the process left off, the browser makes another request to the server, sending the cursor back as a parameter, and this continues until the batch finishes. This is nice because you can display a progress bar to the user.
Finally, you could have an agent running on the server that waits for batch jobs to be submitted and runs them completely outside of the HTTP request lifecycle. So your browser makes a request to kick off a batch job, which results in some sort of record in a database that can keep track of the status of the job. The agent picks up the job and sets it to a pending state while it works, then sets the completion status when it finishes. You can set set something up that allows you to poll the server from the browser periodically so you can alert the user when the process finishes. Of you can just have the agent send an email report to the user when the batch completes. This is the most solid option with the least risk of something interrupting the processing, and it can leave an audit trail without any effort at all. But it's clearly more complicated to set up.

Answer 2

Thanks Rob.
Your response sent me in the right direction.
I sort of used your backend idea on the frontend. I just looped through 20 records at a time and then refreshed the page via javascript and started at 21 to 40 etc. Threw in a progress bar for fun as well.
Thanks for helping me get my head around the idea. Not the right way to do it, but my Python is just as bad as my PHP.
<?php
#ini_set('zlib.output_compression', 0);
#ini_set('implicit_flush', 1);
set_time_limit(600);
#ob_end_clean();
require("header.php");
require_once('nav.php');
function sync_systems($sfData){
$dataPost = $sfData;
$postID = $dataPost['ID'];
$epcall = update_profile($dataPost);
$epResult = json_decode($epcall, true);
if($epResult['status'] != 404){
$sfStatus = updateSFOpportunity($epResult['client_id'], $epResult['ep_id'] );
if($sfStatus == 1){
$datamsg = " Success! The sync was a success in both Salesforce and other system. OtherSystem Record " . $epResult['ep_id'] . " was created or updated.<br/>";
} else {
$datamsg = " Success! The sync was a success in other system, but failed in Salesforce<br/>";
}
echo json_encode(['code'=>200, 'msg'=>$datamsg]);
} else {
$datamsg = " Failure! The sync did not work.<br/>";
echo json_encode(['code'=>404, 'msg'=>$datamsg]);
} // end epResult
}
function sync_ep($sfData){
$dataPost = $sfData;
$postID = $dataPost['ID'];
$epcall = update_profile($dataPost);
$epResult = json_decode($epcall, true);
if($epResult['status'] != 404){
// $sfStatus = updateSFOpportunity($epResult['client_id'], $epResult['ep_id'] );
if($sfStatus == 1){
$datamsg = " Success! The sync was a success in both Salesforce and other system. Other System Record " . $epResult['ep_id'] . " was created or updated.<br/>";
} else {
$datamsg = " Success! The sync was a success in other system, but failed in Salesforce<br/>";
}
echo json_encode(['code'=>200, 'msg'=>$datamsg]);
} else {
$datamsg = " Failure! The sync did not work.<br/>";
echo json_encode(['code'=>404, 'msg'=>$datamsg]);
} // end epResult
}
$ju = "CustomerData20Fall.json";
//read json file from url in php
$readJSONFile = file_get_contents($ju);
//convert json to array in php
$jfile = json_decode($readJSONFile);
//print_r($jfile);
//convert json to array in php
$epSync = array();
$oldValue = 0;
$total = count($jfile );
?>
<!-- Progress bar holder -->
<div id="progress" style="width:500px;border:1px solid #ccc;"></div>
<!-- Progress information -->
<div id="information" style="width"></div>
<?php
if(isset($_REQUEST["NEXTVALUE"])){
$nextValue = $_REQUEST["NEXTVALUE"];
} else {
$nextValue = 1;
}
$refreshValue = $nextValue + 20;
$displaycounter = $nextValue;
$timeRemaining = 0;
$updatedRecords = 0;
foreach ($jfile as $key => $jsons) {
$newKey = $key;
if($oldValue != $newKey){
if($newKey >= $nextValue && $newKey < $refreshValue){
// echo "Updated: " . [$oldValue]['EPID'] . "<br/>";
// echo "<hr>" . $nextValue . " >= " . $newKey . " < " . $refreshValue;
print_r($epSync[$oldValue]);
$displaycounter = $newKey;
echo sync_systems($epSync[$oldValue]);
usleep(30000);
flush();
} else {
if($key == ($refreshValue)){
$theURL = "sf-ep-sync.php?NEXTVALUE=" . $refreshValue . "&RAND=" . rand();
// echo "<hr>" . $newKey . " = " . $refreshValue . " " . $theURL ."<br/>";
echo "<script>location.href = '" . $theURL . "';</script>";
exit;
}
}
$oldValue = $newKey;
$i = $nextValue + 1;
if(($i + 1) == $total ){
$percent = intval($i/$total * 100)."%";
$timeRemaining = 0;
} else {
$percent = intval($i/$total * 100)."%";
$timeRemaining = (($total - $displaycounter)/60);
}
usleep(30000);
echo '<script language="javascript">
document.getElementById("progress").innerHTML="<div style=\"width:'.$percent.';background-color:#ddd;\"> </div>";
document.getElementById("information").innerHTML="'.$displaycounter.' row(s) of '. $total . ' processed. About ' . round($timeRemaining, 2) . ' minutes remaining.";
</script>';
// This is for the buffer achieve the minimum size in order to flush data
echo str_repeat(' ',1024*64);
}
foreach($jsons as $key => $value) {
$epSync[$newKey][$key] = $value;
}
}
Thanks,
Jason