How can I trim white space off image with Intervention? - php

I have a class that takes photos I download before reuploading to S3 for long-term storage and I want to trim the white space. When I store the temporary image (which works fine), I then try to call the Intervention trim() on it in my storeTempFile() method but that doesn't trim the white space. I'm thinking it could be the placement in my code?
class PhotoProcessor
{
public function __construct(Listing $listing, $photoData)
{
$this->bucket = 'real-estate-listings';
$this->s3 = App::make('aws')->get('s3');
$this->tempFileName = 'app/storage/processing/images/retsphotoupload';
$this->photoData = $photoData;
$this->listing = $listing;
$this->photo = new RetsPhoto;
}
public function process()
{
$this->storeTempFile();
$this->storeFileInfo();
$this->buildPhoto();
$success = $this->pushToS3();
// if Result has the full URL or you want to build it, add it to $this->photo
DB::connection()->disableQueryLog();
$this->listing->photos()->save($this->photo);
$this->removeTempFile();
unset ($this->photoData);
return $success;
}
private function storeTempFile()
{
// return File::put($this->tempFileName, $this->photoData['Data']) > 0;
File::put($this->tempFileName, $this->photoData['Data']);
Image::make($this->tempFileName)->trim('top-left', null, 60);
return($this->tempFileName) > 0;
}
private function storeFileInfo()
{
$fileInfo = getimagesize($this->tempFileName);
// Could even be its own object
$this->fileInfo = [
'width' => $fileInfo[0],
'height' => $fileInfo[1],
'mimetype' => $fileInfo['mime'],
'extension' => $this->getFileExtension($fileInfo['mime'])
];
}
private function buildPhoto()
{
$this->photo->number = $this->photoData['Object-ID']; // Storing this because it is relevant order wise
$this->photo->width = $this->fileInfo['width'];
$this->photo->height = $this->fileInfo['height'];
$this->photo->path = $this->getFilePath();
}
private function getFilePath()
{
$path = [];
if ($this->listing->City == NULL)
{
$path[] = Str::slug('No City');
}
else
{
$path[] = Str::slug($this->listing->City, $separator = '-');
}
if ($this->listing->Subdivision->subdivision_display_name == NULL)
{
$path[] = Str::slug('No Subdivision');
}
else
{
$path[] = Str::slug($this->listing->Subdivision->subdivision_display_name, $separator = '-');
}
if ($this->listing->MLSNumber == NULL)
{
$path[] = Str::slug('No MLSNumber');
}
else
{
$path[] = Str::slug($this->listing->MLSNumber, $separator = '-');
}
$path[] = $this->photoData['Object-ID'].'.'.$this->fileInfo['extension'];
return strtolower(join('/', $path));
}
private function pushToS3()
{
return $this->s3->putObject([
'Bucket' => $this->bucket,
'Key' => $this->photo->path,
'ContentType'=> $this->fileInfo['mimetype'],
'SourceFile' => $this->tempFileName
]);
}
private function getFileExtension($mime)
{
// Use better algorithm than this
$ext = str_replace('image/', '', $mime);
return $ext == 'jpeg' ? 'jpg' : $ext;
}
private function removeTempFile()
{
return File::delete($this->tempFileName);
}
}

trim() returns the trimmed image, but you are doing nothing with the return value. Try calling save() to persist the image to the filesystem:
Image::make($this->tempFileName)->trim('top-left', null, 60)->save();
// ^^^^^^

Related

Laravel Model Removing html attribute

Created a simple miniCMS in a portal for content creation. The issue at first was in TinyMCE stripping of id attribute from html tag I've resolved that using valid_elements now the request is being sent to Model as is with no issues however in the Model level it's stripping the id again
Example
<div id="agreement">text ......... </div>
Being Saved in model as
<div>text ......... </div>
The controller code:
public function frontendContent(Request $request, $key)
{
$purifier = new \HTMLPurifier();
$valInputs = $request->except('_token', 'image_input', 'key', 'status', 'type');
foreach ($valInputs as $keyName => $input) {
if (gettype($input) == 'array') {
$inputContentValue[$keyName] = $input;
continue;
}
$inputContentValue[$keyName] = $purifier->purify($input);
}
$type = $request->type;
if (!$type) {
abort(404);
}
$imgJson = #getPageSections()->$key->$type->images;
$validation_rule = [];
$validation_message = [];
foreach ($request->except('_token', 'video') as $input_field => $val) {
if ($input_field == 'has_image' && $imgJson) {
foreach ($imgJson as $imgValKey => $imgJsonVal) {
$validation_rule['image_input.'.$imgValKey] = ['nullable','image','mimes:jpeg,jpg,png,svg'];
$validation_message['image_input.'.$imgValKey.'.image'] = inputTitle($imgValKey).' must be an image';
$validation_message['image_input.'.$imgValKey.'.mimes'] = inputTitle($imgValKey).' file type not supported';
}
continue;
}elseif($input_field == 'seo_image'){
$validation_rule['image_input'] = ['nullable', 'image', new FileTypeValidate(['jpeg', 'jpg', 'png'])];
continue;
}
$validation_rule[$input_field] = 'required';
}
$request->validate($validation_rule, $validation_message, ['image_input' => 'image']);
if ($request->id) {
$content = Frontend::findOrFail($request->id);
} else {
$content = Frontend::where('data_keys', $key . '.' . $request->type)->first();
if (!$content || $request->type == 'element') {
$content = Frontend::create(['data_keys' => $key . '.' . $request->type]);
}
}
if ($type == 'data') {
$inputContentValue['image'] = #$content->data_values->image;
if ($request->hasFile('image_input')) {
try {
$inputContentValue['image'] = uploadImage($request->image_input,imagePath()['seo']['path'], imagePath()['seo']['size'], #$content->data_values->image);
} catch (\Exception $exp) {
$notify[] = ['error', 'Could not upload the Image.'];
return back()->withNotify($notify);
}
}
}else{
if ($imgJson) {
foreach ($imgJson as $imgKey => $imgValue) {
$imgData = #$request->image_input[$imgKey];
if (is_file($imgData)) {
try {
$inputContentValue[$imgKey] = $this->storeImage($imgJson,$type,$key,$imgData,$imgKey,#$content->data_values->$imgKey);
} catch (\Exception $exp) {
$notify[] = ['error', 'Could not upload the Image.'];
return back()->withNotify($notify);
}
} else if (isset($content->data_values->$imgKey)) {
$inputContentValue[$imgKey] = $content->data_values->$imgKey;
}
}
}
}
$content->update(['data_values' => $inputContentValue]);
$notify[] = ['success', 'Content has been updated.'];
return back()->withNotify($notify);
}
When I dd the request
as dd($request) I can see the html tag in full
<div id="agreement">text ......... </div>
But when I dd the content
as dd($content) I can see that the id attribute is stripped
<div>text ......... </div>
The model part
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Model;
class Frontend extends Model
{
protected $guarded = ['id'];
protected $table = "frontends";
protected $casts = [
'data_values' => 'object'
];
public static function scopeGetContent($data_keys)
{
return Frontend::where('data_keys', $data_keys);
}
}
Kindly asking for help, thank you!
While checking the forum here at SOF I found a solution with a remark from #FarhanIbnWahid thanks to him.
$config = HTMLPurifier_Config::createDefault();
$config->set('Attr.EnableID', true);
$purifier = new \HTMLPurifier($config);
This will resolve the issue.

Add target _blank to external link - Parsedown PHP

I'm using Parsedown to parse HTML from the database to my site. With Parsedown, you can't really add target="_blank" to the links.
So what I'm trying to do is to add target="_blank" to external links. I've found this function in Parsedown.php:
protected function inlineLink($Excerpt)
{
$Element = array(
'name' => 'a',
'handler' => 'line',
'text' => null,
'attributes' => array(
'href' => null,
'title' => null,
),
);
$extent = 0;
$remainder = $Excerpt['text'];
if (preg_match('/\[((?:[^][]++|(?R))*+)\]/', $remainder, $matches))
{
$Element['text'] = $matches[1];
$extent += strlen($matches[0]);
$remainder = substr($remainder, $extent);
}
else
{
return;
}
if (preg_match('/^[(]\s*+((?:[^ ()]++|[(][^ )]+[)])++)(?:[ ]+("[^"]*"|\'[^\']*\'))?\s*[)]/', $remainder, $matches))
{
$Element['attributes']['href'] = $matches[1];
if (isset($matches[2]))
{
$Element['attributes']['title'] = substr($matches[2], 1, - 1);
}
$extent += strlen($matches[0]);
}
else
{
if (preg_match('/^\s*\[(.*?)\]/', $remainder, $matches))
{
$definition = strlen($matches[1]) ? $matches[1] : $Element['text'];
$definition = strtolower($definition);
$extent += strlen($matches[0]);
}
else
{
$definition = strtolower($Element['text']);
}
if ( ! isset($this->DefinitionData['Reference'][$definition]))
{
return;
}
$Definition = $this->DefinitionData['Reference'][$definition];
$Element['attributes']['href'] = $Definition['url'];
$Element['attributes']['title'] = $Definition['title'];
}
$Element['attributes']['href'] = str_replace(array('&', '<'), array('&', '<'), $Element['attributes']['href']);
return array(
'extent' => $extent,
'element' => $Element,
);
}
Now, what I've tried is this (added a comment of what I changed):
protected function inlineLink($Excerpt)
{
$Element = array(
'name' => 'a',
'handler' => 'line',
'text' => null,
'attributes' => array(
'href' => null,
'target' => null, // added this
'title' => null,
),
);
$extent = 0;
$remainder = $Excerpt['text'];
if (preg_match('/\[((?:[^][]++|(?R))*+)\]/', $remainder, $matches))
{
$Element['text'] = $matches[1];
$extent += strlen($matches[0]);
$remainder = substr($remainder, $extent);
}
else
{
return;
}
if (preg_match('/^[(]\s*+((?:[^ ()]++|[(][^ )]+[)])++)(?:[ ]+("[^"]*"|\'[^\']*\'))?\s*[)]/', $remainder, $matches))
{
$Element['attributes']['href'] = $matches[1];
if (isset($matches[2]))
{
$Element['attributes']['title'] = substr($matches[2], 1, - 1);
}
$extent += strlen($matches[0]);
}
else
{
if (preg_match('/^\s*\[(.*?)\]/', $remainder, $matches))
{
$definition = strlen($matches[1]) ? $matches[1] : $Element['text'];
$definition = strtolower($definition);
$extent += strlen($matches[0]);
}
else
{
$definition = strtolower($Element['text']);
}
if ( ! isset($this->DefinitionData['Reference'][$definition]))
{
return;
}
$Definition = $this->DefinitionData['Reference'][$definition];
$Element['attributes']['href'] = $Definition['url'];
if (strpos($Definition['url'], 'example.com') !== false) { // added this aswell, checking if its our own URL
$Element['attributes']['target'] = '_blank';
}
$Element['attributes']['title'] = $Definition['title'];
}
$Element['attributes']['href'] = str_replace(array('&', '<'), array('&', '<'), $Element['attributes']['href']);
return array(
'extent' => $extent,
'element' => $Element,
);
}
Any suggestions to accomplish this?
Ran into this issue today. I wanted to have all links from a different host open up in a new target automatically. Unfortunately, the accepted answer recommends editing the Parsedown class file, which is a bad idea imo.
I created a new PHP class which extends Parsedown, and created an override for the element method. Here is the whole class:
class ParsedownExtended extends Parsedown
{
protected function element(array $Element)
{
if ($this->safeMode) {
$Element = $this->sanitiseElement($Element);
}
$markup = '<' . $Element['name'];
if (isset($Element['name']) && $Element['name'] == 'a') {
$server_host = isset($_SERVER['HTTP_HOST']) ? $_SERVER['HTTP_HOST'] : null;
$href_host = isset($Element['attributes']['href']) ? parse_url($Element['attributes']['href'], PHP_URL_HOST) : null;
if ($server_host != $href_host) {
$Element['attributes']['target'] = '_blank';
}
}
if (isset($Element['attributes'])) {
foreach ($Element['attributes'] as $name => $value) {
if ($value === null) {
continue;
}
$markup .= ' ' . $name . '="' . self::escape($value) . '"';
}
}
if (isset($Element['text'])) {
$markup .= '>';
if (!isset($Element['nonNestables'])) {
$Element['nonNestables'] = array();
}
if (isset($Element['handler'])) {
$markup .= $this->{$Element['handler']}($Element['text'], $Element['nonNestables']);
}
else {
$markup .= self::escape($Element['text'], true);
}
$markup .= '</' . $Element['name'] . '>';
}
else {
$markup .= ' />';
}
return $markup;
}
}
Here is where the magic happens:
if (isset($Element['name']) && $Element['name'] == 'a') {
$server_host = isset($_SERVER['HTTP_HOST']) ? $_SERVER['HTTP_HOST'] : null;
$href_host = isset($Element['attributes']['href']) ? parse_url($Element['attributes']['href'], PHP_URL_HOST) : null;
if ($server_host != $href_host) {
$Element['attributes']['target'] = '_blank';
}
}
Now I simply use ParsedownExtended instead of Parsedown when parsing content, e.g.:
$parsedown = new ParsedownExtended();
return $parsedown->text($this->body);
Hope this helps someone.
Such issue already exists on GitHub. Please see this comment.
My extension can automatically set rel="nofollow" and target="_blank"
attributes to a link when it is detected as an external link. You can
also set those attributes manually through the attribute block:
[text](http://example.com) {rel="nofollow" target="_blank"}
Automatic rel="nofollow" Attribute on External Links
// custom external link attributes
$parser->links_external_attr = array(
'rel' => 'nofollow',
'target' => '_blank'
);
If you want to make changes in Parsedown class without using the parsedown-extra-plugin extension, you can do as follows:
1) In \Parsedown::element method after the first line $markup = '<'.$Element['name']; add this line $Element = $this->additionalProcessElement($Element);
2) Add new method to Parsedown class:
protected function additionalProcessElement($Element) { }
3) Extend Parsedown class and save it as MyParsedown.php file:
<?php
namespace myapps;
require_once __DIR__.'/Parsedown.php';
/**
* Class MyParsedown
* #package app
*/
class MyParsedown extends \Parsedown
{
/**
* #param array $Element
* #return array
*/
protected function additionalProcessElement($Element)
{
if ($Element['name'] == 'a' && $this->isExternalUrl($Element['attributes']['href'])) {
$Element['attributes']['target'] = '_blank';
}
return $Element;
}
/**
* Modification of the funciton from answer to the question "How To Check Whether A URL Is External URL or Internal URL With PHP?"
* #param string $url
* #param null $internalHostName
* #see https://stackoverflow.com/a/22964930/7663972
* #return bool
*/
protected function isExternalUrl($url, $internalHostName = null) {
$components = parse_url($url);
$internalHostName = ($internalHostName == null) ? $_SERVER['HTTP_HOST'] : $internalHostName;
// we will treat url like '/relative.php' as relative
if (empty($components['host'])) {
return false;
}
// url host looks exactly like the local host
if (strcasecmp($components['host'], $internalHostName) === 0) {
return false;
}
$isNotSubdomain = strrpos(strtolower($components['host']), '.'.$internalHostName) !== strlen($components['host']) - strlen('.'.$internalHostName);
return $isNotSubdomain;
}
}
4) Create test.php file and run it:
require_once __DIR__.'/MyParsedown.php';
$parsedown = new \myapps\MyParsedown();
$text = 'External link to [example.com](http://example.com/abc)';
echo $parsedown->text($text);
This HTML code will be displayed on the browser page (if your host is not example.com, of course):
<p>External link to example.com</p>
Just like kjdion84 I'd also extend the Parsedown class. I suggest to not copy and change the element method but overwrite inlineLink; it's less work and more future proof if the base code changes.
Heads up: the urlIsExternal method is by no means complete (host check is missing).
class ParsedownExtended extends Parsedown
{
protected function inlineLink($Excerpt)
{
$link = parent::inlineLink($Excerpt);
if ($this->urlIsExternal($link['element']['attributes']['href'])) {
$link['element']['attributes'] += [
'target' => '_blank',
'rel' => 'nofollow',
];
}
return $link;
}
protected function urlIsExternal($url)
{
$scheme = parse_url($url, PHP_URL_SCHEME);
$host = parse_url($url, PHP_URL_HOST);
if (!$scheme || !$host) {
return false;
}
if (strpos(strtolower($scheme), 'http') !== 0) {
return false;
}
// #TODO check the host
return true;
}
}
This will work.
<?php
declare(strict_types=1);
namespace YourNamespace;
class ParsedownExt extends \Parsedown
{
// Add target to links
protected function element(array $Element)
{
if (strcasecmp($Element['name'], 'a')===0)
$Element['attributes']['target'] = '_blank';
return parent::element($Element);
}
}

ValidatorPluginManager get fails to do its job in model input filter

I have a question about Abstract Validators. I was trying to implement the solution of Mb Rostami found here.
This is the error I get:
Zend\Validator\ValidatorPluginManager::get was unable to fetch or create an instance for Application\Validators\File\Image
All I need to do I guess is to somehow inject the class into the model. What is Application\Validators\File\Image?
So how to fix this error? Most easy solution would be to add the validator class as an invocable to the module?
The input filter in model class:
public function getInputFilter()
{
if (!$this->inputFilter) {
$inputFilter = new InputFilter();
$inputFilter->add(array(
'name' => 'eid',
'required' => true,
'filters' => array(
array('name' => 'Int'),
)
));
$newFileName = sha1(time(), true);
$inputFilter->add(
array(
'name' => 'ImageValidator',
'required' => true,
'validators' => array(
array(
'name' => '\Application\Validators\File\Image',
'options' => array(
'minSize' => '64',
'maxSize' => '5120',
'newFileName' => $newFileName,
'uploadPath' => './data/'
),
),
)
)
);
$this->inputFilter = $inputFilter;
}
return $this->inputFilter;
}
Validator class:
<?php
namespace Application\Validators\File;
use Zend\Validator\File\Extension;
use Zend\File\Transfer\Adapter\Http;
use Zend\Validator\File\FilesSize;
use Zend\Filter\File\Rename;
use Zend\Validator\File\MimeType;
use Zend\Validator\AbstractValidator;
class Image extends AbstractValidator
{
const FILE_EXTENSION_ERROR = 'invalidFileExtention';
const FILE_NAME_ERROR = 'invalidFileName';
const FILE_INVALID = 'invalidFile';
const FALSE_EXTENSION = 'fileExtensionFalse';
const NOT_FOUND = 'fileExtensionNotFound';
const TOO_BIG = 'fileFilesSizeTooBig';
const TOO_SMALL = 'fileFilesSizeTooSmall';
const NOT_READABLE = 'fileFilesSizeNotReadable';
public $minSize = 64; //KB
public $maxSize = 1024; //KB
public $overwrite = true;
public $newFileName = null;
public $uploadPath = './data/';
public $extensions = array('jpg', 'png', 'gif', 'jpeg');
public $mimeTypes = array(
'image/gif',
'image/jpg',
'image/png',
);
protected $messageTemplates = array(
self::FILE_EXTENSION_ERROR => "File extension is not correct",
self::FILE_NAME_ERROR => "File name is not correct",
self::FILE_INVALID => "File is not valid",
self::FALSE_EXTENSION => "File has an incorrect extension",
self::NOT_FOUND => "File is not readable or does not exist",
self::TOO_BIG => "All files in sum should have a maximum size of '%max%' but '%size%' were detected",
self::TOO_SMALL => "All files in sum should have a minimum size of '%min%' but '%size%' were detected",
self::NOT_READABLE => "One or more files can not be read",
);
protected $fileAdapter;
protected $validators;
protected $filters;
public function __construct($options)
{
$this->fileAdapter = new Http();
parent::__construct($options);
}
public function isValid($fileInput)
{
$options = $this->getOptions();
$extensions = $this->extensions;
$minSize = $this->minSize;
$maxSize = $this->maxSize;
$newFileName = $this->newFileName;
$uploadPath = $this->uploadPath;
$overwrite = $this->overwrite;
if (array_key_exists('extensions', $options)) {
$extensions = $options['extensions'];
}
if (array_key_exists('minSize', $options)) {
$minSize = $options['minSize'];
}
if (array_key_exists('maxSize', $options)) {
$maxSize = $options['maxSize'];
}
if (array_key_exists('newFileName', $options)) {
$newFileName = $options['newFileName'];
}
if (array_key_exists('uploadPath', $options)) {
$uploadPath = $options['uploadPath'];
}
if (array_key_exists('overwrite', $options)) {
$overwrite = $options['overwrite'];
}
$fileName = $fileInput['name'];
$fileSizeOptions = null;
if ($minSize) {
$fileSizeOptions['min'] = $minSize * 1024;
}
if ($maxSize) {
$fileSizeOptions['max'] = $maxSize * 1024;
}
if ($fileSizeOptions) {
$this->validators[] = new FilesSize($fileSizeOptions);
}
$this->validators[] = new Extension(array('extension' => $extensions));
if (!preg_match('/^[a-z0-9-_]+[a-z0-9-_\.]+$/i', $fileName)) {
$this->error(self::FILE_NAME_ERROR);
return false;
}
$extension = pathinfo($fileName, PATHINFO_EXTENSION);
if (!in_array($extension, $extensions)) {
$this->error(self::FILE_EXTENSION_ERROR);
return false;
}
if ($newFileName) {
$destination = $newFileName . ".$extension";
if (!preg_match('/^[a-z0-9-_]+[a-z0-9-_\.]+$/i', $destination)) {
$this->error(self::FILE_NAME_ERROR);
return false;
}
} else {
$destination = $fileName;
}
$renameOptions['target'] = $uploadPath . $destination;
$renameOptions['overwrite'] = $overwrite;
$this->filters[] = new Rename($renameOptions);
$this->fileAdapter->setFilters($this->filters);
$this->fileAdapter->setValidators($this->validators);
if ($this->fileAdapter->isValid()) {
$this->fileAdapter->receive();
return true;
} else {
$messages = $this->fileAdapter->getMessages();
if ($messages) {
$this->setMessages($messages);
foreach ($messages as $key => $value) {
$this->error($key);
}
} else {
$this->error(self::FILE_INVALID);
}
return false;
}
}
}
The solution was simple, at least in my case. Like I mentioned above: check the folder structure. Zend Framework has it's own way of structuring project files, so the file path needs to match.

Getting a fatal Error call to unidentified method Pass::_createpass()

Can someone please help. I am getting a fatal error on the following PHP script.
I am getting an error "unidentified method Pass::_createpass()" whick relates to the second last line of the code below.
<?php
$engine = Pass::start('xxxxxxxxxxxxxxxx');
$pass = $engine->createPassFromTemplate(xxxxxxxxxxxxxx);
$engine->redirectToPass($pass);
if (!function_exists('curl_init')) {
throw new Exception('Pass needs the CURL PHP extension.');
}
if (!function_exists('json_decode')) {
throw new Exception('Pass needs the JSON PHP extension.');
}
$engine = Pass::start($appKey);
$values = array(
'first' => 'John',
'last' => 'Platinum',
);
$images = array(
'thumbnail' => 'image1.jpg'
);
$pass = $engine->createPassFromTemplate(5688667418918912, $values, $images);
$passData = $engine->downloadPass($pass);
$engine->redirectToPass($pass);
class Pass
{
private $_appKey = null;
private $_endpoint = 'https://pass.center/api/v1';
private $_debug = false;
private static $_instance = null;
private static $_imageTypes = array('icon', 'logo', 'strip', 'thumbnail', 'background', 'footer');
const VERSION = '0.5';
const USER_AGENT = 'PassSDK-PHP/0.5';
public function __construct($appKey = null, $endpoint = null, $debug = false)
{
if (is_null($appKey)) {
throw new Exception('App Key required');
}
$this->_appKey = $appKey;
if ($endpoint !== null) {
$this->_endpoint = $endpoint;
}
$this->_debug = $debug;
}
public static function start($appKey = null, $endpoint = null, $debug = false)
{
if (self::$_instance == null) {
self::$_instance = new self($appKey, $endpoint, $debug);
}
return self::$_instance;
}
public function createPassFromTemplate($templateId, $values = array(), $images = array())
{
$resource = sprintf("https://xxxxxxx/api/v1/templates/names/Test/pass", $templateId);
return $this->_createPass($resource, $values, $images);
}
}
I hope someone can help me as I am not familiar with Functions PHP.
Thanks All
Rob
The function createPass is missing.
You need something like this:
/**
* Prepares the values and image for the pass and creates it
*
* #param string $resource Resource URL for the pass creation
* #param array $values Values
* #param array $images Images
* #return object Pass
*/
private function _createPass($resource, $values, $images)
{
$multipart = count($images) > 0;
if ($multipart) {
$content = array();
foreach ($images as $imageType => $image) {
$this->_addImage($image, $imageType, $content, $imageType);
}
var_dump($content);
// Write json to file for curl
$jsonPath = array_search('uri', #array_flip(stream_get_meta_data(tmpfile())));
file_put_contents($jsonPath, json_encode($values));
$content['values'] = sprintf('#%s;type=application/json', $jsonPath);
} else {
$content = $values;
}
return $this->_restCall('POST', $resource, $content, $multipart);
}
..And full script here

Why is my app running out of memory? Unsetting variables, using chunk

I have the simple app below. I'm turning off query logging in Laravel, I'm unsetting where possible, yet this function will only process about 800 records before I'm out of RAM on my 2GB Linode. I know I'm asking a lot of you guys but I can't seem to see where I'm leaking memory.
There are really only two major steps.
Step 1 - Move records from a temp table to production
class ListingMigrator
{
public function __construct($tempListing, $feed)
{
$this->tempListing = $tempListing;
$this->listing = $this->listingInstance();
$this->feed = $feed;
}
public static function migrateListing($listing, $feed)
{
$instance = new static($listing, $feed);
return $instance->migrate();
}
public function migrate()
{
$this->addExternalData();
$this->populateListing();
$this->processPhotos();
$this->deleteTempListing();
}
private function listingInstance()
{
DB::connection()->disableQueryLog();
$listing = Listing::findByMud($this->tempListing->matrix_unique_id);
return $listing ?: new Listing;
}
private function processPhotos()
{
$retsApi = new RetsFeedApi($this->feed);
/* Initialize Object */
$rets = $retsApi->findMostRecent();
$photos = $rets->getPhotosForListing($this->listing->matrix_unique_id);
foreach ($photos as $photo)
{
$uploader = new PhotoProcessor($this->listing, $photo);
$uploader->process();
}
}
private function populateListing()
{
DB::connection()->disableQueryLog();
$this->listing->fill($this->tempListing->toArray());
$this->listing->imported_at = $this->tempListing->created_at;
$this->listing->board = $this->tempListing->board;
return $this->listing->save();
}
private function addExternalData()
{
// Get Google lattitude and longitude
$googlecoords = getGoogleMapInfo($this->tempListing->FullAddress, $this->tempListing->City);
$this->listing->GoogleLat = $googlecoords['GoogleLat'];
$this->listing->GoogleLong = $googlecoords['GoogleLong'];
// Add or update the Subdivision Table (helper function)
$subdivisiondata = SubdivisionUpdate($this->tempListing->board, $this->tempListing->SubCondoName, $this->tempListing->Development);
$this->listing->SubdivisionID = $subdivisiondata['id'];
}
private function deleteTempListing()
{
return $this->tempListing->delete();
}
}
Step 2 - Download photos and reupload to Amazon S3
class PhotoProcessor
{
public function __construct(Listing $listing, $photoData)
{
$this->bucket = 'real-estate-listings';
$this->s3 = App::make('aws')->get('s3');
$this->tempFileName = 'app/storage/processing/images/retsphotoupload';
$this->photoData = $photoData;
$this->listing = $listing;
$this->photo = new RetsPhoto;
}
public function process()
{
$this->storeTempFile();
$this->storeFileInfo();
$this->buildPhoto();
$success = $this->pushToS3();
// if Result has the full URL or you want to build it, add it to $this->photo
DB::connection()->disableQueryLog();
$this->listing->photos()->save($this->photo);
$this->removeTempFile();
unset ($this->photoData);
return $success;
}
private function storeTempFile()
{
return File::put($this->tempFileName, $this->photoData['Data']) > 0;
}
private function storeFileInfo()
{
$fileInfo = getimagesize($this->tempFileName);
// Could even be its own object
$this->fileInfo = [
'width' => $fileInfo[0],
'height' => $fileInfo[1],
'mimetype' => $fileInfo['mime'],
'extension' => $this->getFileExtension($fileInfo['mime'])
];
}
private function buildPhoto()
{
$this->photo->number = $this->photoData['Object-ID']; // Storing this because it is relevant order wise
$this->photo->width = $this->fileInfo['width'];
$this->photo->height = $this->fileInfo['height'];
$this->photo->path = $this->getFilePath();
}
private function getFilePath()
{
$path = [];
if ($this->listing->City == NULL)
{
$path[] = Str::slug('No City');
}
else
{
$path[] = Str::slug($this->listing->City, $separator = '-');
}
if ($this->listing->Development == NULL)
{
$path[] = Str::slug('No Development');
}
else
{
$path[] = Str::slug($this->listing->Development, $separator = '-');
}
if ($this->listing->Subdivision == NULL)
{
$pathp[] = Str::slug('No Subdivision');
}
else
{
$path[] = Str::slug($this->listing->Subdivision, $separator = '-');
}
if ($this->listing->MLSNumber == NULL)
{
$pathp[] = Str::slug('No MLSNumber');
}
else
{
$path[] = Str::slug($this->listing->MLSNumber, $separator = '-');
}
$path[] = $this->photoData['Object-ID'].'.'.$this->fileInfo['extension'];
return strtolower(join('/', $path));
}
private function pushToS3()
{
return $this->s3->putObject([
'Bucket' => $this->bucket,
'Key' => $this->photo->path,
'ContentType'=> $this->fileInfo['mimetype'],
'SourceFile' => $this->tempFileName
]);
}
private function getFileExtension($mime)
{
// Use better algorithm than this
$ext = str_replace('image/', '', $mime);
return $ext == 'jpeg' ? 'jpg' : $ext;
}
private function removeTempFile()
{
return File::delete($this->tempFileName);
}
}
Edit to show RetsPhoto
class RetsPhoto extends Eloquent {
protected $table = 'rets_property_photos';
public function listing() {
return $this->belongsTo('Listing', 'matrix_unique_id', 'matrix_unique_id');
}
}
Edit #2: Chunk Call
This is in the app/command and the only thing in there is the fire() function below:
public function fire()
{
// Turn off query logging
DB::connection()->disableQueryLog();
$feeds = RetsFeed::where('active','=',1)->get();
foreach ($feeds as $feed)
{
$class = "TempListing{$feed->board}";
$listings = $class::orderBy('MatrixModifiedDT','desc');
$listings->chunk(50, function($listings) use($feed) {
$listings->each(function($listing) use ($feed) {
ListingMigrator::migrateListing($listing,$feed);
echo "Feed: $feed->board\r\n";
echo "SubcondoName: $listing->SubCondoName\r\n";
echo "Development: $listing->Development\r\n";
echo "\r\n";
});
});
}
}
I think I have figured it out.
Your system holds in memory all of the photo data. As witnessed by the unset ($this->photoData);
The problem is that you need to first complete the process function. Your application is not likely processing ANY photos so when you keep grabbing them from the file system you run out of memory BEFORE you even process a single one.
To Confirm this, simply grab 1 file not using the chunk method.
I am not very familar with Laravel, it could be grabbing all of the files all at once as well and eating the ram.
You can do some tracing with memory_get_usage(true) to find out exactly where the ram is getting eaten from. I would suggest analysing the fire method first.

Categories