We recently started our first TYPO3 10 project and are currently struggling with a custom import script that moves data to Algolia. Basically, everything works fine, but there is an issue with FAL images, specifically, when they need to be processed.
From the logs, I could find something called DeferredBackendImageProcessor, but the docs are not mentioning this, or I am not looking for the right thing. I'm not sure.
Apparently, images within the backend environment are not just processed anymore. There is something called "processingUrl" which has to be called once for the image to be processed.
I tried calling that url with CURL, but it does not work. The thing is, when I open that "processingUrl" in the browser, it has not effect - but if I open that link in a browser, where I am logged into the TYPO3 backend, then the image is processed.
I'm kind of lost here, as I need the images to be processed within the import script that runs via the scheduler from the backend (manual, not via cron).
That is the function where the problem occurs, the curl part has no effect here, sadly.
protected function processImage($image, $imageProcessingConfiguration)
{
if ($image) {
$scalingOptions = array (
'width' => 170
);
$result = $this->contentObject->getImgResource('fileadmin/'.$image, $scalingOptions);
if (isset($result[3]) && $result[3]) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $result[3]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
return '/fileadmin'.$result['processedFile']->getIdentifier();
}
}
return '';
}
$result[3] being the processing url. Example of the url:
domain.com/typo3/index.phproute=%2Fimage%2Fprocess&token=6cbf8275c13623a0d90f15165b9ea1672fe5ad74&id=141
So my question is, how can I process the image from that import script?
I am not sure if there is a more elegant solution but you could disable the deferred processing during your jobs:
$processorConfiguration = $GLOBALS['TYPO3_CONF_VARS']['SYS']['fal']['processors']
unset ($GLOBALS['TYPO3_CONF_VARS']['SYS']['fal']['processors']['DeferredBackendImageProcessor'])
// ... LocalImageProcessor will be used
$GLOBALS['TYPO3_CONF_VARS']['SYS']['fal']['processors'] = $processorConfiguration;
References:
https://github.com/TYPO3/TYPO3.CMS/blob/10.4/typo3/sysext/core/Classes/Resource/Processing/ProcessorRegistry.php
https://github.com/TYPO3/TYPO3.CMS/blob/10.4/typo3/sysext/core/Configuration/DefaultConfiguration.php#L284
Related
I want to use the amphp/parallel library for non-blocking process. I have a simple download file function which does a curl hit to the remote image file and save it to the local. I'm hitting this method through a REST API. Basically I want a process where aysnc download should be done on backend and it could be said as, REST API hit the function and function says "Hey, OK I'm downloading in background you can proceed ahead". Means non-blocking and API gets response as ok , not to wait. Meanwhile, if there is some network failure onto download, worker can restart the process in some time. How do I start?
I have tried the following code, but did not work.
require_once "vendor/autoload.php";
use Amp\Loop;
use Amp\Parallel\Worker\CallableTask;
use Amp\Parallel\Worker\DefaultWorkerFactory;
\Amp\Loop::run(function () {
$remote_file_url = "some remote image url"; //http://example.com/some.png
$file_save_path = "save path for file"; //var/www/html/some.png
$factory = new DefaultWorkerFactory();
$worker = $factory->create();
$result = yield $worker->enqueue(new CallableTask('downloadFile', [$remote_file_url, $file_save_path]));
$code = yield $worker->shutdown();
});
//downloadFile is a simple download function
function downloadFile($remoteFile, $localFile) {
if (!$remoteFile || !$localFile) {
return;
}
set_time_limit(0);
$fp = fopen($localFile, 'w+');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $remoteFile);
curl_setopt($ch, CURLOPT_TIMEOUT, 50);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$result = curl_exec($ch);
curl_close($ch);
fclose($fp);
return $result ? true : false;
}
I'm getting this error:
PHP Fatal error: Uncaught Amp\\Parallel\\Worker\\TaskError: Uncaught Error in worker with message "Call to undefined function downloadFile()" and code "0" in /var/www/html/test/vendor/amphp/parallel/lib/Worker/Internal/TaskFailure.php:45\nStack trace:\n#0 /var/www/html/test/vendor/amphp/parallel/lib/Worker/TaskWorker.php(126): Amp\\Parallel\\Worker\\Internal\\TaskFailure->promise()\n#1 [internal function]: Amp\\Parallel\\Worker\\TaskWorker->Amp\\Parallel\\Worker\\{closure}()\n#2 /var/www/html/test/vendor/amphp/amp/lib/Coroutine.php(76): Generator->send(Object(Amp\\Parallel\\Worker\\Internal\\TaskFailure))\n#3 /var/www/html/test/vendor/amphp/amp/lib/Internal/Placeholder.php(130): Amp\\Coroutine->Amp\\{closure}(NULL, Object(Amp\\Parallel\\Worker\\Internal\\TaskFailure))\n#4 /var/www/html/test/vendor/amphp/amp/lib/Coroutine.php(81): Amp\\Coroutine->resolve(Object(Amp\\Parallel\\Worker\\Internal\\TaskFailure))\n#5 /var/www/html/test/vendor/amphp/amp/lib/Internal/Placeholder.php(130): Amp\\Coroutine->Amp\\{closure}(NULL, Object(Amp\\Parallel\\Worker\\Internal\\TaskFailur in /var/www/html/test/vendor/amphp/parallel/lib/Worker/Internal/TaskFailure.php on line 45
I have similar requirement as asked in How does amphp work regarding the background running process.
Generally, Amp doesn't work magically in the background. If you use PHP via PHP-FPM or alike Amp will be shut down once the response is done, just like anything else.
If you want to move work from these requests into background processes, you need some kind of queue (e.g. beanstalkd) and a (permanent) worker to process these queued jobs. You can write such a daemonized worker with Amp, but it will have to be started out-of-band.
That said, if you just want concurrent downloads amphp/artax is better suited than using amphp/parallel, as it has a way lower overhead compared to a separate PHP process per HTTP request.
The question doesn't clarify where the downloadFile() function has been defined. As per amphp/parallel documentation, the callback must be auto-loaded so that Amphp can find it when the task is executed.
Here's a suggestion:
Put the downloadFile() function in a separate file, say functions.inc.
In your composer.json, under autoload/files, add an entry for functions.inc.
{
"autoload": {
"files": ["functions.inc"]
}
}
Run composer install so that the autoload.php is regenerated to reflect the above change.
Try executing the file containing your first code snippet containing Loop::run() etc.
I think this should do the trick. Apart from this, please refer to kelunik's comment which contains valuable information.
I need some help Using the Mediawiki API with the "Continue" or "query-continue" Command to pull information from my wiki articles. I have a large number of wiki articles (more than 800 currently) and I need to use the api to pull them in batches of 50 and then print ou sections.
My API call works properly:
//Stackoverflow making me use a valid URL here, this api is actually my own localhost server
http://en.wikipedia.org/w/api.php?action=query&list=allpages&apfrom=a&apto=z&apnamespace=0&format=xml&aplimit=50 I am querying all pages, therefore "apfrom" and "apto".
I just need help processing the code with PHP and CURL accessing the API and processing the batches of 50 and using the "continue" to access more records until I hit the end. So far my php code is:
//the CURL commands here work and outputs a data set but only for the first 50 records, so I need to call "continue" to get to the end.
//My api url is localhost but I'm forced to use a valid URL by Stackoverflow.com
$url = sprintf('http://en.wikipedia.org/w/api.php?
action=query&list=allpages&apfrom=a&apto=z&apnamespace=0&format=xml&aplimit=50');
$ch=curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'My site');
$res = curl_exec($ch);
$continue = '';
while ( // I don't know what to set here as true to get the while loop going, maybe continue = true? maybe set query-continue as true?)
{
//Maybe I need something other than $res['query-continue]??
if (empty($res['query-continue']))
{
exit;
}
else
{
$continue = '&apcontinue='.urlencode($res['query-continue']);
foreach ($res['query']['allpages'] as $v)
{
echo $v['title'];
}
}
}
Can someone correct my while loop code above so I can do a simple print out of the title from each wiki article in the loop? I've done a lot of searching online but I'm stuck!! I found a python loop example at http://www.mediawiki.org/wiki/API:Query but I have to do it in PHP. And I am not sure if I call continue or query-continue.
As svick said, please use a client library which handles continuation for you.
The query continuation mechanism has changed multiple times in MediaWiki, you don't want to understand it or even less rely on it.
i've been tasked converting a payment gateway for magento from asp to php, the asp one is outdated for an old site and what's required is a new one, looking at all of the asp code i can easily do it but there is one thing that gets me.
Magento uses this Varien_Http_Adapter_Curl to send xml to the payment gateway and while i understand how the .asp file get the xml data (using Request.InputStream) i can't seem to duplicate that in a php file. this is as close as i can get
function Page_Load()
{
$cUrl = curl_init();
$dump = curl_exec($cUrl);
$file = fopen("Gateway.txt","w");
echo fwrite($file,var_export($dump,true));
fclose($file);
var_dump($dump);
}
Page_Load();
i changed the payment gateway URL in magento backend to my php file and went though the checkout, it creates the .txt file but all it contains is false
so how do i receive the output from magento in my Page_Load function, at the moment i'm just outputting it to a file just to confirm that i am getting a response
UPDATE: i have chnaged the code to this
function Page_Load()
{
$cUrl = curl_init();
curl_setopt($cUrl,CURLOPT_URL,"http://my.site.local/Current-Build/gateway/Gateway.php");
curl_setopt($cUrl,CURLOPT_RETURNTRANSFER,true);
$dump = curl_exec($cUrl);
if(!$dump)
{
$dump = curl_error($cUrl);
}
$file = fopen("Gateway.txt","a");
echo fwrite($file,var_export($dump,true));
fclose($file);
curl_close($cUrl);
var_dump($dump);
}
Page_Load();
the site is on a wamp served that's been put online so anyone on in my development team can access it and it is also the same path that is specified in magento in the payment methods for the payment gateway url which was set to the old aspx file
when i then go though the checkout, while magento gives me the normal error cause it's not getting a response my output file isn't even being created, when i try to go back to magento it just sits there loading and i have to restart WAMP Server a couple of times cause it sits on the orange logo for ages
I'll start with what my program does. The index function of controller takes an array of URLs and keywords and stores them in DB. Now the crawlLink method with take all the keywords and URLs. The URLs are searched for all the keywords and the sublinks of all the URLs are generated and again stored in DB which are also searched for the keywords. Keywords are searched in each link using search method. The sublinks are extracted from all the URLs using extract_links function. search and extract_links both have a method called get_web_page which takes the complete content of the page using cURL. get_web_page is used once in search function to get content of web page so that keywords can be extracted from it. It is also used in extract_links function to extract links with valid page content.
Now crawlLink calls search function twice. Once to extract keywords from domain links and second time to extract keywords from sublinks. Hence, get_web_page is called thrice. It approximately takes 5 mins to get contents of around 150 links. And it is called thrice so it takes 15 minutes of processing time. During that duration nothing can be done. Thus, I want to run this process in background and show its status while processing. extract_links and get_web_page are included in the controller using include_once.
The get_web_page function is as follows:
function get_web_page( $url )
{
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle compressed
CURLOPT_USERAGENT => "spider", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
}
An input of URLs and keywords once from the user can be considered as a task. Now this task can be started and it will start running in the background. At the same time another task can be defined and can be started. Each task will have statuses like "To Do", "In Progress", "Pending", "Done", etc. The Simple Task Board by Oscar Dias is the exact way I want the tasks to be displayed.
I read about so many ways to run function in background that now I am in a dilemma about which approach to adopt. I read about exec, pcntl_fork, Gearman and other but all need CLI which I don't want to use. I tried installing Gearman with Cygwin but got stuck in Gearman installation as it cannot find libevent. I've installed libevent separately but still it doesn't work. And Gearman needs CLI so dropped it. I don't want to use CRON also. I just want to know which approach will be best in my scenario.
I am using PHP 5.3.8 | Codeigniter 2.1.3 | Apache 2.2.21 | MySQL 5.5.16 | Windows 7 64 bit
Your problem is, Windows.
windows is simply not very good for running background tasks & cron jobs - there are tools you can find, but they are limited.
However, are you sure you even need this? Most servers are Linux, so why don't you just test on Windows & move over.
--
The second part is command line - you need it if you want to start a new process (which you do). But ti isn't really very scary. CodeIgniter is quite simple:
http://ellislab.com/codeigniter/user-guide/general/cli.html
You can run using nohup process or using cron job.............Please go through below links
nohup: run PHP process in background
Running a php5 background process under Linux
https://nsaunders.wordpress.com/2007/01/12/running-a-background-process-in-php/
The above approach that I was trying to achieve didn't seem possible to be implemented in Windows. Many methods listed in the questions are either removed or modified. I then moved on to a workaround involving use of AJAX.
I execute the controller method as an ajax request and give a count to it which increments with each new AJAX request. Each request can be aborted though the processing will continue but ultimately results matter in my project even if they are taken incomplete. And if the browser is open then that request may complete and later on the user can see the complete result.
On stopping the processing of a task a CANCELLED icon is shown and a link pointing to result page is shown which displays the results generated before the task was cancelled. On AJAX fails or AJAX success I send back the count of the task from server to client which was sent by the client to server. Thus results are displayed for a unique task and don't get messed up.
But there is no tracking of how much a certain task has progressed. The time taken for execution cannot be identified. Thus, this approach works for me but has some drawbacks. The main aim was the user should not be waiting while some task is in progress and that is somehow achieved by the above workaround.
I will set up a register page using MSSQL.
The system must work like:
User appends data at something.com/register.php
The data is sent to host-ip-address/regsecond.php which my database will be at. (For security reasons, this php page wont directly access to the database.
The php page at host will start another PHP page or EXE file will directly reach database directly and securely.
As my php level is not high, I wanted to learn If i could start php scripts which will work and do their job without coming into users browsers. Here I explain what I say:
" I append some data at x.php, and it starts another PHP script which will do the job with the DATA appended from x.php but the -another PHP script- wont come into users browser "
I was hopefully clear ,as summary, should I use exe [will be harder] or can I start PHP script without coming into browser. And how of course.
You can do this using the curl extension. You can find info on it here:
http://php.net/manual/en/book.curl.php
You can do something like the following:
$postdata = array(
'item1' => 'data'
);
$ch = curl_init("http://host-ip-address/regsecond.php");
curl_setopt ($ch, CURLOPT_POST, true);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_exec($ch);
curl_close($ch);
This makes a call directly from your first script to your second script without exposing anything to the user. On the far side, the data will come in as regular post data ($_POST).
You can't post data through PHP to a different website.
If you would like your website then you can configure your PHP script to connect to a different server for your MySQL, I wouldn't say it's a huge amount safer. For example
Instead of:
mysql_connect(localhost,username,password);
Try this
mysql_connect(http://your-ip:portnumber,username,password);
I'm not sure I understand this correctly but you may
§1 use a "public" php script that invokes a private one:
<?php
//public register script
//now call private
//store data to txt-file or similar..
require('/path/outside/www-data/script_that_processes_further.php');
§2 request a script at another server,
<?php
file_get_contents('http://asdf.aspx?firstname=' . $theFirstName); //simplistic
//other options would be curl, xml/soap or whatever.
§1 may be used with §2.
regards,
/t