mongodb update exception 'document fragment is too large' - php

Updating my collection record field with 'MongoBinData', exception is triggered:
"document fragment is too large: 21216456, max: 16777216"
I find some web discussion about 'allowDiskUse:true' for aggregate, but nothing ubout 'update'.
Here a part of code in PHP:
try {
$criteria = array( '_id' => $intReleaseId);
$fileData = file_get_contents( $_FILES[ $fileKey]["tmp_name"]);
$mongoBinData = new MongoBinData( $fileData, MongoBinData::GENERIC)
$docItem['data'] = $mongoBinData;
$docItem['fileType'] = $strFileType;
$docItem['fileSize'] = $intFileSize;
$docItem['fileExtension'] = $strFileExtension;
$docItem['fileName'] = $strFileName;
$options = array( "upsert" => true,
'safe' => true, 'fsync' => true,
'allowDiskUse' => true ); // this option doesn't change anything
$reportJson = self::GetCollection('releases')->update( $criteria, $docItem, $options);
...
MongoDb release is db version v3.0.6
Some idea ?

Self resolved.
Use gridFS is immediate and simple.
$_mongo = new MongoClient();
$_db = $_mongo->selectDB($_mDbName);
$_gridFS = $_db->getGridFS();
/* */
function saveFileData($intReleaseId, $binData) {
$criteria = array( '_id' => $intReleaseId);
// if exist or not, remove previous value
try {
$_gridFS->remove( $criteria);
}
catch(Exception $e) {}
// store new file content
$storeByteCompleted = false;
try {
$reportId = $_gridFS->storeBytes(
$binData,
array("_id" => $intReleaseId));
if ($reportId == $intReleaseId) {
$storeByteCompleted = true;
}
catch(Exception $e) {}
return $storeByteCompleted;
}
function loadFileData($intReleaseId) {
$gridfsFile = null;
$binData = null;
try {
$gridfsFile = $_gridFS->get($intReleaseId);
}
catch(Exception $e) {}
if ($gridfsFile != null) {
$binData = $gridfsFile->getBytes()
}
return $binData;
}
That's all.

Related

Problem of Curly Brackets in my controller Php Symfony

I want to call my function but when I call it I have a problem with curly Brackets at the end of my code and i have this error Error SYMFONY ( {} ) in my Controller.
I have no idea where to put them for my code to work. I have this problem when I add my function that allows me to retrieve the
history of the action. The mentioned function goes as this:
$this->logHistory->addHistoryConnection($project->getId(), $user->getId(), 'Delete Local Suf', $sf_code);
Function Supp Suf
/**
* #Route("/creation/suf/supp", name="suf_supp")
*/
public function suf(
Request $request,
ShapesRepository $shapesRepository
) {
$params = $this->requestStack->getSession();
$projet = $params->get('projet');
$modules = $params->get('modules');
$fonctionnalites = $params->get('fonctionnalites');
$user = $this->getUser()->getUserEntity();
$manager = $this->graceManager;
$mapManager = $this->mapManager;
$countElements = $mapManager->getCount();
$shapes = $shapesRepository->findBy(array('projet' => $projet->getId()));
$adresseWeb = $this->getParameter('adresse_web');
$carto = $params->get('paramCarto');
$centrage = $params->get('centrage');
$cableColor = $params->get('cableColor');
$sf_code = '';
if ($request->get('suf') != '') {
$sf_code = $request->get('suf');
}
$suf = $manager->getSuf($sf_code);
$success = '';
$error = '';
$warning = '';
if ($request->query->get('success')) {
$success = $request->query->get('success');
} elseif ($request->query->get('error')) {
$error = $request->query->get('error');
} elseif ($request->query->get('warning')) {
$warning = $request->query->get('warning');
}
if ($request->isMethod('POST')) {
if ($request->request->get('sf_code') != '') {
$sf_code = $request->request->get('sf_code');
}
if ($request->get('val') != '') {
$val = $request->get('val');
}
$dir = $this->getparameter('client_directory');
$dossier = str_replace(' ', '_', $projet->getProjet());
$dir = $dir . $dossier . '/documents/';
$cable = $val[0];
$chem = $val[1];
$t_suf = $this->graceCreator->supprimeSuf($sf_code, $cable, $chem);
if ($t_suf[0][0] == '00000') {
$this->logHistorique->addHistoryConnection($projet->getId(), $user->getId(), 'Suppression Suf Local', $sf_code);
// $creator->delDirObjet( $st_code, $dir );
$data = new JsonResponse(array("success" => "create!"));
return $data;
} else {
$data = new JsonResponse(array("error" => "Error : " . $t_suf));
return $data;
}
return $this->render('Modifications/supSuf.html.twig', array(
'user' => $user,
'paramCarto' => $carto,
'cableColor' => $cableColor,
'suf' => $suf,
'adresseWeb' => $adresseWeb,
'centrage' => $centrage,
'shapes' => $shapes,
'projet' => $projet,
'modules' => $modules,
'fonctionnalites' => $fonctionnalites,
'countElements' => $countElements
));
}
}
Your only return statement is inside of an if condition. If the code does not pass the condition, it has nothing to return. The code must return something in all possible cases. If you are not still used to these practices, an IDE might guide you until it becomes a routine. PHPStorm is my personal preference.
BTW, I recommend you to switch from the array() syntax to the more globally accepted [] although you must be in PHP 5.4 or higher.

List Azure files and folders from File share snapshots with php

To list the files and folders from Azure Files can be done with this code:
function ListFolder($shareName, $path)
{
global $fileRestProxy;
$vMaxResultados = 5000;
$vNextMarker = "";
$listResult = null;
try
{
do
{
$options = new ListDirectoriesAndFilesOptions();
$options->setMaxResults($vMaxResultados);
$options->setMarker($vNextMarker);
$listResult = $fileRestProxy->ListDirectoriesAndFiles($shareName,$path,$options);
$vNextMarker = $listResult->getNextMarker();
} while ($vNextMarker != "");
}
catch (Exception $e)
{
$code = $e->getCode();
$error_message = $e->getMessage();
return "ERROR:$code:$error_message";
}
return $listResult;
}
But how is the sintaxis or method to the same with a snapshot from these Share?
This doesn't work:
function ListSnapshotFolder($shareName, $path, $snapshot)
{
global $fileRestProxy;
$vMaxResultados = 5000;
$vNextMarker = "";
$listResult = null;
try
{
do
{
$options = new ListDirectoriesAndFilesOptions();
$options->setMaxResults($vMaxResultados);
$options->setMarker($vNextMarker);
$shareFull = $shareName . "?snapshot=" . $snapshot;
$listResult = $fileRestProxy->ListDirectoriesAndFiles($shareFull,$path,$options);
$vNextMarker = $listResult->getNextMarker();
} while ($vNextMarker != "");
}
catch (Exception $e)
{
$code = $e->getCode();
$error_message = $e->getMessage();
return "ERROR:$code:$error_message";
}
return $listResult;
}
Is there any parameter in the $option object to add?
Or maybe the $shareFull must be created in some format?
$shareFull = $shareName . "?snapshot=" . $snapshot;
Thanks in advance.
I believe you have found a bug in the SDK. I looked up the source code here and there's no provision to provide sharesnapshot query string parameter in the options as well as the code does not even handle it.
public function listDirectoriesAndFilesAsync(
$share,
$path = '',
ListDirectoriesAndFilesOptions $options = null
) {
Validate::notNull($share, 'share');
Validate::canCastAsString($share, 'share');
Validate::canCastAsString($path, 'path');
$method = Resources::HTTP_GET;
$headers = array();
$postParams = array();
$queryParams = array();
$path = $this->createPath($share, $path);
if (is_null($options)) {
$options = new ListDirectoriesAndFilesOptions();
}
$this->addOptionalQueryParam(
$queryParams,
Resources::QP_REST_TYPE,
'directory'
);
$this->addOptionalQueryParam(
$queryParams,
Resources::QP_COMP,
'list'
);
$this->addOptionalQueryParam(
$queryParams,
Resources::QP_PREFIX_LOWERCASE,
$options->getPrefix()
);
$this->addOptionalQueryParam(
$queryParams,
Resources::QP_MARKER_LOWERCASE,
$options->getNextMarker()
);
$this->addOptionalQueryParam(
$queryParams,
Resources::QP_MAX_RESULTS_LOWERCASE,
$options->getMaxResults()
);
$this->addOptionalQueryParam(
$queryParams,
Resources::QP_TIMEOUT,
$options->getTimeout()
);
$dataSerializer = $this->dataSerializer;
return $this->sendAsync(
$method,
$headers,
$queryParams,
$postParams,
$path,
Resources::STATUS_OK,
Resources::EMPTY_STRING,
$options
)->then(function ($response) use ($dataSerializer) {
$parsed = $dataSerializer->unserialize($response->getBody());
return ListDirectoriesAndFilesResult::create(
$parsed,
Utilities::getLocationFromHeaders($response->getHeaders())
);
}, null);
}
You may want to open up an issue here: https://github.com/Azure/azure-storage-php/issues and bring this to SDK team's attention.

solr data can not be sent before solrcommit

What my problem is that I can not send array to solr machine in order to update. I am using codeigniter as a framework and here is my code:
$solrData = array();
$solrData['id'] = $this->data['profil_data']['id'];
$solrData['site'] = $this->data['profil_data']['site'];
$solrData['url_Domain'] = $this->data['profil_data']['url_Domain'];
$solrData['url_Page'] = $this->data['profil_data']['url_Page'];
$solrData['url_Profil'] = $this->data['profil_data']['url_Profil'];
$solrData['scr_Kobi_Rank'] = $this->data['profil_data']['scr_Kobi_Rank'];
$solrData['scr_A'] = $this->data['profil_data']['scr_A'];
$solrData['scr_B'] = $this->data['profil_data']['scr_B'];
$solrData['scr_C'] = $this->data['profil_data']['scr_C'];
$solrData['scr_D'] = $this->data['profil_data']['scr_D'];
$solrData['loc_City'] = $this->input->post('plakano');
$solrData['loc_Lat_Lon'] = $this->input->post('loc_Lat_Lon');
$solrData['com_Category'] = explode(',', $this->input->post('category'));
$urunData = $this->input->post('urun_list');
foreach($urunData as $row)
{
$ontoData = $this->m_onto->getOntoDataByOntoDataId($row);
$solrData['com_Products'][] = $ontoData['baslik'];
}
$hizmetData = $this->input->post('hizmet_list');
foreach($hizmetData as $row)
{
$ontoData = $this->m_onto->getOntoDataByOntoDataId($row);
$solrData['com_Services'][] = $ontoData['baslik'];
}
$solrData['com_Type'] = $this->input->post('sirketturu');
$solrData['com_Description'] = $this->input->post('description');
$solrData['com_Title_Selected'] = $this->input->post('title');
$solrData['com_Title_Long'] = $this->data['profil_data']['com_Title_Long'];
$solrData['crm_Tel'] = $this->input->post('tel');
$solrData['crm_Fax'] = $this->input->post('fax');
$solrData['crm_Email'] = $this->input->post('email');
$this->solr->updateSolrProfilData($solrData);
And solr process:
public function updateSolrProfilData($arrData)
{
if(count($arrData) == 0)
return FALSE;
$solrClientOptions = $this->solrClientOptionsYazProfil;
$solrClientOptionsCommit = $this->solrClientOptionsYazProfilCommit;
$solrClient = new SolrClient($solrClientOptions);
$solrDoc = new SolrInputDocument();
foreach($arrData as $firmaField => $firmaValue)
{
if(! is_array($firmaValue))
{
$solrDoc->addField($firmaField, $firmaValue);
}
else
{
foreach($firmaValue as $firmaField2 => $firmaValue2)
{
if($firmaValue2 != '')
{
$solrDoc->addField($firmaField, $firmaValue2);
}
}
}
}
try {
$this->_solrCommit($solrClientOptionsCommit);
} catch (Exception $e) {
echo $e->getMessage();
}
}
Solr Commit function:
private function _solrCommit($solrArr)
{
$urlCommit = 'http://' . $solrArr['hostname'] . ":" . $solrArr['port'] . '/' . $solrArr['path'] . "/update?stream.body=%3Ccommit/%3E&wt=json";
$output = file_get_contents($urlCommit);
$outputArr = json_decode($output, TRUE);
if ($outputArr['responseHeader']['status'] === 0)
return TRUE;
else
return FALSE;
}
And that is the options:
private $solrClientOptionsYazProfilCommit = array(
'hostname' => SOLR_HOST_YAZ,
'login' => '',
'password' => '',
'port' => SOLR_PORT,
'path' => 'solr/collection1'
);
Altough try-catch returns no error, the data can not be updated. Moreover, code sends solr commit succesfully. I checked the url but it is in correct form. What is wrong in here?
Dont use PHP/Pecl solr libs. If you can access solr via a URL then you should just use PHP and CURL:
static function doCurl($url, $username = null, $password = null) {
if (!function_exists('curl_init')) {
// throw error
}
$ch = curl_init();
$opts = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_POST => 1,
CURLOPT_TIMEOUT => 120,
CURLOPT_FAILONERROR => 1,
CURLOPT_HTTPAUTH => CURLAUTH_ANY
);
if ($password != null && $username != null) {
$opts[CURLOPT_USERPWD] = "$username:$password";
}
curl_setopt_array($ch, $opts);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
usage is:
doCurl("http://hostNameHere:8983/solr/select/?q=solr&start=0&rows=10&indent=on", "user", "pass");
Your issue is that you are never issuing a command to Solr to add the document that you have built to your index. You are only issuing the commit command, which is executing successfully.
Since you are using PHP, I would recommend using the PHP SolrClient. This will save you from having to manually write all of the functions (add, delete, commit, etc.) yourself. In this case, you would need to call the addDocument function.

reduce php daemon memory usage

could you please help me to find what cause this process to reach 500MB of memory usage.
It is basically an html page downloader.
Despite the fact that the process is stable (and do not exceed that limit), it' meant to use on low performing machine and I'm not satisfied.
The size of the mysql table 'Sites' is 170MB.
following the script code.
Thanks in advance.
function start() {
try {
global $log;
$db = getConnection();
Zend_Db_Table::setDefaultAdapter($db);
$log->logInfo("logger start");
while (1) {
$sitesTable = new Zend_Db_Table('Sites');
$rowset = $sitesTable->fetchAll();
foreach ($rowset as $row) {
if (time() >= (strtotime($row->lastUpdate) + $row->pollingHours * 60 * 60)) {
db_updateHtml($row);
}
}
}
} catch (Exception $e) {
global $log;
$log->logError($e->getMessage());
}
}
function db_updateHtml($siteRecord) {
try {
if ($siteRecord instanceof Zend_Db_Table_Row) {
$rowwithConnection = $siteRecord;
$url = $siteRecord->url;
$idSite = $siteRecord->idSite;
$crawler = new Crawler();
$sitesTable = new Zend_Db_Table('Sites');
//$rowwithConnection = $sitesTable->fetchRow(
// $sitesTable->select()->where('idSite = ?', $idSite));
$newHtml = HtmlDbEncode($crawler->get_web_page($url));
if (strlen($newHtml) < 10) {
global $log;
$log->logError("Download failed for: url: $url \t idsite: $idSite ");
}
if ($rowwithConnection->isChecked != 0) {
$rowwithConnection->oldHtml = $rowwithConnection->newHtml;
$rowwithConnection->isChecked = 0;
}
$rowwithConnection->newHtml = $crawler->get_web_page($url);
$rowwithConnection->lastUpdate = date("Y-m-d H:i:s");
//$rowwithConnection->diffHtml = getDiff($rowwithConnection->oldHtml, $rowwithConnection->newHtml, false, $rowwithConnection->minLengthChange);
$rowwithConnection->diffHtml = getDiffFromRecord($rowwithConnection, false, $rowwithConnection->minLengthChange);
/* if (strlen($rowwithConnection->diffHtml) > 30) {
$rowwithConnection->lastChanged = $rowwithConnection->lastUpdate;
} */
$rowwithConnection->save();
} else {
$log->logCrit("siteRecord is uninitialized");
}
} catch (Exception $e) {
global $log;
$log->logError($e->getMessage());
}
}
function getDiffFromRecord($row, $force = false, $minLengthChange = 100) {
if ($row instanceof Zend_Db_Table_Row) {
require_once '/var/www/diff/library/finediff.php';
include_once '/var/www/diff/library/Text/Diff.php';
$diff = new AndreaDiff();
$differences = $diff->getDiff($row->oldHtml, $row->newHtml);
if ($diff->isChanged($minLengthChange) || $force) {
$row->lastChanged = $row->lastUpdate;
$row->isChecked = false;
return ($differences);
}
}
return null;
}
function getConnection() {
try {
$pdoParams = array(
PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => true
);
$db = new Zend_Db_Adapter_Pdo_Mysql(array(
'host' => '127.0.0.1',
'username' => 'root',
'password' => 'administrator',
'dbname' => 'diff',
'driver_options' => $pdoParams
));
return $db;
} catch (Exception $e) {
global $log;
$log->logError($e->getMessage());
}
}
1) Try use fetch method, not fetchAll:
foreach($sitesTable->fetch() as $row){
//...
}
2) try to unset all variables which store html code (if you save it in memory), at last iteration i suppose variable $rowwithConnection will have html code inside.
When i want profile php application i use xhprof it will save you a LOT of time. Good Luck!

Zend framework $db->update result

Zend_Db_Adapter::update() returns the number of rows affected by the update operation.
What is best way to determine if the query was successful?
$data = array(
'updated_on' => '2007-03-23',
'bug_status' => 'FIXED'
);
$n = $db->update('bugs', $data, 'bug_id = 2');
$data = array(
'updated_on' => '2007-03-23',
'bug_status' => 'FIXED',
);
$n = 0;
try {
$n = $db->update('bugs', $data, 'bug_id = 2');
} catch (Zend_Exception $e) {
die('Something went wrong: ' . $e->getMessage());
}
if (empty($n)) {
die('Zero rows affected');
}
If you are just looking for a boolean return value, this solution is the best for handling success in the model:
class Default_Model_Test extends Zend_Db_Table {
public function updateTest($testId, $testData){
try {
$this->_db->update('test', $testData, array(
'id = ?' => $testId
));
return true;
}
catch (Exception $exception){
return false;
}
}
}
But, a better solution would be to handling success from the controller level, because it is making the request:
class Default_Model_Test extends Zend_Db_Table {
public function updateTest($testId, $testData){
$this->_db->update('test', $testData, array(
'id = ?' => $testId
));
}
}
class Default_TestController extends Zend_Controller_Action {
public function updateAction(){
try {
$testId = $this->_request->getParam('testId');
if (empty($testId)) throw new Zend_Argument_Exception('testId is empty');
$testData = $this->_request->getPost();
if (empty($testId)) throw new Zend_Argument_Exception('testData is empty');
$testModel->updateTest($testId, $testData);
}
catch (Exception $exception){
switch (get_class($exception)){
case 'Zend_Argument_Exception': $message = 'Argument error.'; break;
case 'Zend_Db_Statement_Exception': $message = 'Database error.'; break;
case default: $message = 'Unknown error.'; break;
}
}
}
}
This is an excellent solution when working with multiple resource types, using a switch on the exception type, and doing what is appropriate based on your program's needs. Nothing can escape this vacuum.
Maybe:
$data = array(
'updated_on' => '2007-03-23',
'bug_status' => 'FIXED'
);
$result = $db->update('bugs', $data, 'bug_id = 2');
if ($result < $numRows) {//pass in numRows as method arg or hardcode integer.
//handle error
} else {
return TRUE;
}
Try something like this with the idea being you want to verify that the number of records you wanted updated got updated.

Categories