Multiple file downloading using Guzzle (Pool) doesn't writes any file - php

I need to download multiple files with Guzzle using concurrent requests so i choose to use Pool.
Basically, I wrote a function that receives a list of URL that are the remote files i need to download and a path which is the directory where i want to save the files.
The function should save in the provided directory each file I'm sending the request to but instead the directory reamins empty and no file as been wrote inside it.
What am I doing wrong?
Here is my function with some echos for debugging purpose:
function async_multiple_files_download($files_url_list, $path) {
$client = new \GuzzleHttp\Client();
$requests = array();
for ($i = 0; $i < sizeof($files_url_list); $i++) {
$file_name = basename($files_url_list[$i]);
$request_destination_file_path = $path . DIRECTORY_SEPARATOR . $file_name;
$requests[$i] = new GuzzleHttp\Psr7\Request('GET', $files_url_list[$i], ['sink' => $request_destination_file_path]);
echo "Downloading " . basename($files_url_list[$i]) . "<br>from($files_url_list[$i])<br>to $request_destination_file_path" . "<br><br><br>";
}
$pool = new \GuzzleHttp\Pool($client, $requests, [
'concurrency' => 10,
'fulfilled' => function (\Psr\Http\Message\ResponseInterface $response, $index) {
echo 'success: '.$response->getStatusCode()."<br>";
},
'rejected' => function ($reason, $index) {
echo 'failed: '.$reason."<br>";
},
]);
$promise = $pool->promise();
$promise->wait();
}

Related

Async http call with php

I have a situation where I have a loop, that is going to read chunk of data from a file, send those chunk to a rest api, and continue until the EOF, but I want this to be async inside the loop, so, I don't have to wait until the API respond to read the next chunk.
I have been looking at Amphp and ReactPHP for I can't find a solution to this, or maybe I don't understand how are those libraries supposed to be used.
here is a pseudo of what I am doing.
<?php
while($file.read()){
$chunk = getNextChunk();
sendChunkAsync($chunk);
}
function getNextChunk(){
echo "reading next chunk";
// read next chunk of data
}
sample with amphp
function sendChunkAsync($chunk){
Loop::run(function () {
$uri = "https://testapi.com/api";
$client = new DefaultClient;
try {
$promises = $client->request($uri);
$responses = yield $promises;
echo "chunk processed";
} catch (Amp\Artax\HttpException $error) {
// log error
// $error->getMessage() . PHP_EOL;
}
});
}
In this case I would expect (if reading chunk is faster than getting response from api) something like this, don't take this literary, I am trying to illustrate it for you.
Reading next chunk
Reading next chunk
chunk processed
Reading next chunk
chunk processed
chunk processed
I am going to use React as I know the library better but they work in similar ways.
EDIT: updated, see comments
This will read in a file and every time it recieves a chunk of data, it will create an api call and send the data off
<?php
require_once __DIR__ . '/vendor/autoload.php';
function async_send($config, $file, callable $proccessor)
{
$config['ssl'] = true === $config['ssl'] ? 's' : '';
$client = new \GuzzleHttp\Client([
'base_uri' => 'http' . $config['ssl'] . '://' . $config['domain'] . '/rest/all/V1/',
'verify' => false,
'http_errors' => false
]);
$loop = \React\EventLoop\Factory::create();
$filesystem = \React\Filesystem\Filesystem::create($loop);
$filesystem->getContents($file)->then(function($contents) use ($config, $proccessor, $client) {
$contents = $proccessor($contents);
$client->post($config['uri'], ['body' => $contents]);
});
}
$config = [
'domain' => 'example.com',
'ssl' => true
];
//somewhere later
$configp['uri'] = 'products';
async_send($configp, __DIR__ . 'my.csv', function ($contents) {
return json_encode($contents);
});
In case someone else is trying to solve a similar problem
<?php
require_once __DIR__ . '/../vendor/autoload.php';
use React\HttpClient\Client as ReactClient;
function async_send($loop, $filePath, callable $proccessor)
{
echo "starting";
echo "\n\r";
try {
$filesystem = \React\Filesystem\Filesystem::create($loop);
$file = $filesystem->file($filePath);
$file->open('r')
->then(function ($stream) use ($loop, $proccessor){
$stream->on('data', function ($chunk) use ($loop, $proccessor) {
$proccessor($chunk);
});
});
} catch (\Exception $e) {
echo "failed";
echo "\n\r";
}
echo "ending reading";
echo "\n\r";
}
function callApiReal($loop, $fileChunk = null)
{
echo "ready to call api". PHP_EOL;
$uri = "https://testapi.com/";
try {
$client = new ReactClient($loop);
} catch (\Exception $e) {
echo "Error";
}
echo "ready to call api";
$request = $client->request('POST', $uri, $fileChunk);
$request->on('response', function ($response) use ($uri) {
$response->on('data', function ($data_chunk) {
echo 'data chunk from api received';
echo "\n\r";
});
// subscribe to listen to the end of the response
$response->on('end', function () use ($uri) {
echo "operation has completed";
echo "\n\r";
});
});
$request->on('error', function ($error) {
// something went bad in the request
echo "Damm!";
echo "\n\r";
});
$request->end();
}
// main loop
$loop = React\EventLoop\Factory::create();
//somewhere later
async_send($loop, __DIR__ . '/my.csv', function ($chunk) use ($loop) {
echo "calling api";
callApiReal($loop, $chunk);
echo "\n\r";
});
$loop->run();

Laravel - Pass uploaded filename to new function

I'm using Laravel 5.3 and need to upload an xml file and then submit the contents to an api. The client wants it as 2 buttons/user functions where the user should first upload the file and then with a second click submit the contents.
The uploading is working fine and the xml reading and submitting to api is also working properly. I just can't get my upload controller to pass the filename over to the submitting controller. There is no need to store the filename for future use and the processes will follow each other - ie user will upload one file and submit, then upload next file and submit.
Any help would be highly appreciated
upload function:
public function handleUpload(Request $request)
{
$file = $request->file('file');
$allowedFileTypes = config('app.allowedFileTypes');
$rules = [
'file' => 'required|mimes:'.$allowedFileTypes
];
$this->validate($request, $rules);
$fileName = $file->getClientOriginalName();
$destinationPath = config('app.fileDestinationPath').'/'.$fileName;
$uploaded = Storage::put($destinationPath, file_get_contents($file->getRealPath()));
if($uploaded) {
$file_Name = ($_FILES['file']['name']);
}
return redirect()->to('/upload');
}
submit function:
public function vendorInvoice()
{
$fileName = $file_Name;
$destinationPath = storage_path('app/uploads/');
$xml = file_get_contents($destinationPath.$fileName);
$uri = "some uri";
try {
$client = new Client();
$request = new Request('POST', $uri, [
'Authorization' => '$username',
'ContractID' => '$id',
'content-type' => 'application/xml'
],
$xml);
$response = $client->send($request);
}
catch (RequestException $re) {
//Exception Handling
echo $re;
}
}

Dropbox API stripping MP3 tags

I am synchronizing a Centos directory of MP3's with a Dropbox shared folder. When I copy MP3 files into the folder in Windows all is well. When I upload from Centos using a PHP script and the REST interface the files arrive, but they're a larger size and are missing the tags. I can still play the files so it's not simple file corruption. I'm opening the file in binary mode in the PHP script. Here's the relevant code:
$path = $this->dropboxPath($root, $subDir, $fileName);
$uri = "https://api-content.dropbox.com/1/files_put/auto/$path";
$lclPath = storage_path() . "/$root/$subDir/$fileName";
$fd = fopen($lclPath, 'rb');
$this->putDropbox($uri, [
'overwrite' => 'true'
], $fd
);
private function putDropbox($uri, $parms, $fd) {
$uri = $uri . "?" . http_build_query($parms);
$client = new GuzzleHttp\Client();
$req = $client->createRequest('PUT', $uri, [
'exceptions' => true,
'body' => [
'file_filed' => $fd
]
]);
$req->setHeader('Authorization', 'Bearer ' . $this->token);
try {
$resp = $client->send($req);
return $resp;
}
catch(Exception $e) {
Log::error($e->getRequest());
if($e->hasResponse()) {
Log::error($e->getResponse());
}
}
}

Zend\File\Transfer\Adapter\Http on receive : error "File was not found" with jQuery File Upload

Here is already two questions about this problem
Zf2 file upload by jQuery File Upload - file was not found
Can't get blueimp / jQuery-File-Upload and ZF2 running
without ansers. And i'm create issue on ZF2 with code examples.
github.com/zendframework/zf2/issues/6291
And also have request from another developer on my email with question, how to implement jQuery File Upload with ZF2.
github.com/blueimp/jQuery-File-Upload
So, there is real problem for many peple, and no any manuals, no answers.
Please, before send me to read documentation, notice, that i'm spend many hours on problem and already read all documentation and not only i'm have this problem.
Please, write manual with code examples, like How to implement it. Or just answer, why we have this error and how to resolve it?
There i'm copy my example from ZF2 issue.
I'm try use jQuery-File-Upload
Just copy standard tpl, include css and scrypts and it's work, send files to my controller.
But controller doesn't work.
Here is my code
public function processjqueryAction()
{
$request = $this->getRequest();
$response = $this->getResponse();
$jsonModel = new \Zend\View\Model\JsonModel();
if ($request->isPost()) {
try {
$datas = [];
$datas['files'] = [];
$uploadPath = $this->getFileUploadLocation();
$uploadFiles = $this->params()->fromFiles('files');
// throw new \Exception(json_encode("FILES " . serialize($_FILES)));
// Сохранение выгруженного файла
$adapter = new \Zend\File\Transfer\Adapter\Http();
$adapter->setDestination($uploadPath);
$adapter->setValidators(array(
new \Zend\Validator\File\Extension(array(
'extension' => array('jpg', 'jpeg', 'png', 'rtf')
)
),
// new \Zend\Validator\File\Upload()
));
if (!$adapter->isValid()) {
throw new \Exception(json_encode("!isValid " . implode(" ", $adapter->getMessages())));
}
$files = $adapter->getFileInfo();
// throw new \Exception(json_encode($files));
foreach ($files as $file => $info) {
// throw new \Exception(json_encode($info));
$name = $adapter->getFileName($file);
// file uploaded & is valid
if (!$adapter->isUploaded($file)) {
throw new \Exception(json_encode("!isUploaded") . implode(" ", $adapter->getMessages()));
continue;
}
if (!$adapter->isValid($file)) {
throw new \Exception(json_encode("!isValid " . implode(" ", $adapter->getMessages())));
continue;
}
// receive the files into the user directory
$check = $adapter->receive($file); // this has to be on top
if (!$check) {
throw new \Exception(json_encode("! receive" . implode(" ", $adapter->getMessages())));
}
/**
* "name": "picture1.jpg",
"size": 902604,
"url": "http:\/\/example.org\/files\/picture1.jpg",
"thumbnailUrl": "http:\/\/example.org\/files\/thumbnail\/picture1.jpg",
"deleteUrl": "http:\/\/example.org\/files\/picture1.jpg",
"deleteType": "DELETE"
*/
$fileclass = new stdClass();
// we stripped out the image thumbnail for our purpose, primarily for security reasons
// you could add it back in here.
$fileclass->name = $name;
$fileclass->size = $adapter->getFileSize($name);
$fileclass->type = $adapter->getMimeType($name);
$fileclass->deleteUrl = '/uploads/delete';
$fileclass->deleteType = 'DELETE';
//$fileclass->error = 'null';
$fileclass->url = '/';
$datas['files'][] = $fileclass;
}
$response->getHeaders()->addHeaders(array(
'Pragma' => 'no-cache',
'Cache-Control' => 'private, no-cache',
"Content-Type" => 'application/json'
));
// return $response->setContent(json_encode(array('files' => $files)));
return $response->setContent(json_encode($datas));
} catch (\Exception $e) {
return $response->setContent(json_encode($e->getMessage()));
}
}
return $jsonModel;
}
Sorry for debug code, but with it you can see, i'm try hard to make it work, over 3 hours.
Error is
"File 'CIMG0042.JPG' was not found"
When i'm call $adapter->isValid()
Or when call it with file name, same error.
Path, where files uploaded, is correct and writable. $_FILES array is exist and valid.
Here is $_FILES json
FILES a:1:{s:5:\"files\";a:5:{s:4:\"name\";a:1:{i:0;s:28:\"52876065d17dce0a7472e5d6.jpg\";}s:4:\"type\";a:1:{i:0;s:10:\"image\/jpeg\";}s:8:\"tmp_name\";a:1:{i:0;s:14:\"\/tmp\/phpmfT2mB\";}s:5:\"error\";a:1:{i:0;i:0;}s:4:\"size\";a:1:{i:0;i:82640;}}}
And result of $files = $adapter->getFileInfo();
"{"files_0_":{"name":"52876065d17dce0a7472e5d6.jpg","type":"image\/jpeg","tmp_name":"\/tmp\/phpF6VoO9","error":0,"size":"82640","options":{"ignoreNoFile":false,"useByteString":true,"magicFile":null,"detectInfos":true},"validated":false,"received":false,"filtered":false,"validators":["Zend\\Validator\\File\\Upload","Zend\\Validator\\File\\Extension"],"destination":"\/home\/seyfer\/www\/zend2-tutorial.me\/module\/Users\/config\/..\/..\/..\/data\/uploads"}}"
isUploaded passes, but isValid not.
What i'm doing wrong?
Documentation say this
Zend_File_Transfer has been deprecated in favor of using the standard ZF2 Zend\Form and Zend\InputFilter features.
Maybe it's mean, that Form need to be used for file uploading in any way?
UPD 25.05.14
Now i'm add form
class UploadJqueryForm extends BaseForm
{
public function __construct()
{
parent::__construct(__CLASS__);
$this->setAttribute('method', 'post');
$this->setAttribute('enctype', 'multipart/form-data');
$this->init();
}
public function init()
{
$fileupload = new Element\File('files');
$fileupload->setLabel("files");
$fileupload->setAttribute('multiple', 'multiple');
$this->add($fileupload);
$button = new Element\Button('start');
$button->setAttribute("type", 'submit');
$button->setValue("Start upload")->setLabel("Start upload");
$this->add($button);
$button = new Element\Button('cancel');
$button->setAttribute("type", 'reset');
$button->setValue("Cancel upload")->setLabel("Cancel upload");
$this->add($button);
$button = new Element\Button('delete');
$button->setAttribute("type", 'button');
$button->setValue("Delete")->setLabel("Delete");
$this->add($button);
$checkbox = new Element\Checkbox('toggle');
$checkbox->setValue("Toggle")->setLabel("Toggle");
$checkbox->setAttribute("required", "");
$this->add($checkbox);
}
}
Use it
public function processjqueryAction()
{
$form = new \Users\Form\UploadJqueryForm();
$request = $this->getRequest();
$response = $this->getResponse();
$jsonModel = new \Zend\View\Model\JsonModel();
try {
if ($request->isPost()) {
$data = array_merge_recursive(
$this->getRequest()->getPost()->toArray(), $this->getRequest()->getFiles()->toArray()
);
// throw new \Exception(json_encode("data " . serialize($data)));
$form->setData($data);
if ($form->isValid()) {
$datas = [];
$datas['files'] = [];
$uploadPath = $this->getFileUploadLocation();
// $uploadFiles = $this->params()->fromFiles('files');
// throw new \Exception(json_encode("FILES " . serialize($_FILES)));
// Сохранение выгруженного файла
$adapter = new \Zend\File\Transfer\Adapter\Http();
$adapter->setDestination($uploadPath);
$adapter->setValidators(array(
new \Zend\Validator\File\Extension(array(
'extension' => array('jpg', 'jpeg', 'png', 'rtf')
)
),
));
if (!$adapter->isValid()) {
throw new \Exception(json_encode("!isValid " . implode(" ", $adapter->getMessages())));
}
$files = $adapter->getFileInfo();
// throw new \Exception(json_encode($files));
foreach ($files as $file => $info) {
// throw new \Exception(json_encode($info));
$name = $adapter->getFileName($file);
// file uploaded & is valid
if (!$adapter->isUploaded($file)) {
throw new \Exception(json_encode("!isUploaded") . implode(" ", $adapter->getMessages()));
continue;
}
if (!$adapter->isValid($file)) {
throw new \Exception(json_encode("!isValid " . implode(" ", $adapter->getMessages())));
continue;
}
// receive the files into the user directory
$check = $adapter->receive($file); // this has to be on top
if (!$check) {
throw new \Exception(json_encode("! receive" . implode(" ", $adapter->getMessages())));
}
/**
* "name": "picture1.jpg",
"size": 902604,
"url": "http:\/\/example.org\/files\/picture1.jpg",
"thumbnailUrl": "http:\/\/example.org\/files\/thumbnail\/picture1.jpg",
"deleteUrl": "http:\/\/example.org\/files\/picture1.jpg",
"deleteType": "DELETE"
*/
$fileclass = new stdClass();
// we stripped out the image thumbnail for our purpose, primarily for security reasons
// you could add it back in here.
$fileclass->name = $name;
$fileclass->size = $adapter->getFileSize($name);
$fileclass->type = $adapter->getMimeType($name);
$fileclass->deleteUrl = '/uploads/delete';
$fileclass->deleteType = 'DELETE';
//$fileclass->error = 'null';
$fileclass->url = '/';
$datas['files'][] = $fileclass;
}
$response->getHeaders()->addHeaders(array(
'Pragma' => 'no-cache',
'Cache-Control' => 'private, no-cache',
"Content-Type" => 'application/json'
));
return $response->setContent(json_encode($datas));
} else {
throw new \Exception(json_encode("!isValid form" . serialize($form->getMessages())));
}
}
} catch (\Exception $e) {
return $response->setContent(json_encode($e->getMessage()));
}
return $jsonModel;
And still get error
File '24866-fu-blyad-otvratitelno.jpg' was not found
Also I tried with InputFilter
class UploadJqueryFilter extends InputFilter implements
InputFilterAwareInterface
{
public function __construct()
{
$this->getInputFilter();
}
public function getInputFilter()
{
$toggle = new Input('toggle');
$toggle->setRequired(FALSE);
$this->add($toggle);
$files = new \Zend\InputFilter\FileInput('files');
$files->setRequired(TRUE);
$files->getValidatorChain()->attach(new Validator\File\UploadFile);
$files->getFilterChain()->attach(new \Zend\Filter\File\RenameUpload(array(
'target' => __DIR__ . '/../../../../../../tmpuploads/tmp',
'randomize' => true,
)));
$this->add($files);
return $this;
}
public function setInputFilter(InputFilterInterface $inputFilter)
{
return false;
}
}
And have same error.
I had this problem too. Wasting several hours before found the problem. Turn out it's because the name attribute for input tag cannot be set as 'files'.
So this is a no:
<input id="files" type="file" name="files" data-url="/upload-action" />
Changing the name attribute to any string other than files such as file will solve this problem.
<input id="files" type="file" name="file" data-url="/upload-action" />
I see from $_FILES that you had set name as files. Try changing that.
Make sure you update the reference you made in your controller too.

AWS Multipart upload SDK not working

I need Help on AWS multi-part Upload using PHP 5.3
I am using
1. WAMP Server.
2. PHP 5.3
3. OS: Windows 8
I am trying to upload a Big file to My Amazon S3 bucket.
And tried in many ways. I followed the code procedure from
http://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFilePHP.html
I wrote PHP code but I don't know what mistake I did.
My Code:
<?php
require_once './sdk/sdk.class.php';
use \AmazonS3;
$ufile = $_FILES['ufile'];
$filepath = $ufile['tmp_name'];
$bucket = '**My_bkt**';
var_dump($ufile);
print $keyname = \date("Y") . "/" . \date("F") . "/" . \date("d") . "/" . $ufile['name'] . "<BR />";
$api_key = "***my_api_key***";
$secret_key = "***my_secret_key***";
// Define a megabyte.
define('MB', 1048576);
// Instantiate the class
//$s3 = new AmazonS3();
$s3 = new AmazonS3(array(
'key' => $api_key,
'secret' => $secret_key,
));
// 1. Initiate a new multipart upload.
/* #var $response type */
$response = $s3->initiate_multipart_upload($bucket, $keyname);
// Get the Upload ID.
$upload_id = (string) $response->body->UploadId;
// 2. Upload parts.
// Get part list for a given input file and given part size.
// Returns an associative array.
$parts = $s3->get_multipart_counts(filesize($filepath), 3 * MB);
foreach ($parts as $i => $part) {
// Upload part and save response in an array.
$responses[] = $s3->upload_part($bucket, $keyname, $upload_id, array(
'fileUpload' => $filepath,
'partNumber' => ($i + 1),
'seekTo' => (integer) $part['seekTo'],
'length' => (integer) $part['length'],
)
);
}
// 3. Complete multipart upload. We need all part numbers and ETag values.
$parts = $s3->list_parts($bucket, $keyname, $upload_id);
$response = $s3->complete_multipart_upload($bucket, $keyname, $upload_id, $parts);
var_dump($response);
Please help me.
I know you're using the older SDK 1 here, so my answer doesn't apply directly to what you've posted. That said, SDK 1.x is no longer being updated and SDK 2.x is what all new work should be using (as per the AWS SDK for PHP team).
If you do update your project to use SDK 2, then take a look at S3Client::upload(). It should greatly simplify what you're trying to do here.
I have Updated My SDK and Changed My Code like bellow,
////////////////// AWS Code Begin ////////////////////
/////////////////////////// Step 1 /////////////////////////////
$ufile = $_FILES['Filedata'];
$filename = $ufile['tmp_name'];
$filesize = $ufile['size'];
/* * ************ Calculating Number of Parts ******************* */
$number_of_parts = 0;
$r = $filesize % PART; // Remainder
$q = floor($filesize / PART); // Quotient
if ($r != 0) {
$number_of_parts = $q + 1;
} else {
$number_of_parts = $q;
}
$bucket = 'isource123';
$keyname = date("Y") . "/" . date("F") . "/" . date("d") . "/" . $ufile['name'];
///////////////////////////// Step 2 /////////////////////////////
// Create a service builder using a configuration file
$aws = Aws::factory('./aws/Aws/Common/Resources/aws-config.php');
// Get the client from the builder by namespace
$client = $aws->get('S3');
$uploader = \Aws\S3\Model\MultipartUpload\UploadBuilder::newInstance()
->setClient($client)
->setSource($filename)
->setBucket($bucket)
->setKey($keyname)
->setOption('Metadata', array('Foo' => 'Bar'))
->setOption('CacheControl', 'max-age=3600')
->setConcurrency($number_of_parts)
->build();
try {
$uploader->upload();
echo "Upload complete.\n";
} catch (MultipartUploadException $e) {
$uploader->abort();
echo "Upload failed.\n";
}
I updated My SDK to version 2 and it's working fine.

Categories