Currently I am setting up a queue to insert data from CSV files to the DB using file upload. The queue and data extraction is working currently, but I am looking for a way to check whether the job has been completed, failed or still in process. Is it possible to achieve this and update the status of uploaded file in other DB table as well?
Jobs
Redis::throttle('upload-csv')->allow(1)->every(20)->then(function () {
dump('Processing this file:---',$this->file);
$data = array_map('str_getcsv',file($this->file));
foreach($data as $row){
CSVDataList::updateOrCreate([
'KEY_1' => $row[0],
],[
'KEY_2' => $row[1],
'KEY_3' => $row[2],
'KEY_4' => $row[3],
]);
}
dump('Done for this file:---',$this->file);
unlink($this->file);
}, function () {
return $this->release(10);
});
Controller
$request->validate([
'file' => 'required|mimes:csv,txt'
]);
$filename = $file->getClientOriginalName();
$file = file($request->file->getRealPath());
$data = array_slice($file,1);
$parts = (array_chunk($data,5000));
foreach($parts as $index=>$part){
$fileName = resource_path('pending-files/'.date('y-m-d-H-i-s').$index.'.csv');
file_put_contents($fileName,$part);
}
(new CSVDataList())->importToDb($filename);
session()->flash('status','Queued for importing..');
return redirect('file-upload');
Function in Model
public function importToDb($filename)
{
$path = resource_path('pending-files/*.csv');
$files = glob($path);
foreach($files as $file){
ProcessCsvUpload::dispatch($filename,$file);
}
}
Related
Found a handful of questions on here about this with no answer, so hopefully, someone can point me in the right direction...
I'm trying to create and save a csv file to storage, then update the DB in Laravel. I can create the file successfully, and I can update the DB successfully... but I'm stuck on putting them both together. In my controller, I have this for creating the file (taken from here):
public function updatePaymentConfirm(Request $request) {
$users = User::all();
$fileName = 'test.csv';
$headers = array(
"Content-type" => "text/csv",
"Content-Disposition" => "attachment; filename=$fileName",
"Pragma" => "no-cache",
"Cache-Control" => "must-revalidate, post-check=0, pre-check=0",
"Expires" => "0"
);
$columns = array('First Name', 'Email');
$callback = function() use($users, $columns) {
$file = fopen('php://output', 'w');
fputcsv($file, $columns);
foreach ($users as $user) {
$row['First Name'] = $user->first_name;
$row['Email'] = $user->email;
fputcsv($file, array($row['First Name'], $row['Email']));
}
fclose($file);
};
// return response()->stream($callback, 200, $headers);
}
When the function completes, the last line (that's commented out) prompts the user to download the newly created file (which is not the functionality I'm looking for). I tried adding this to my controller in its place for saving to storage and also updating the database:
$fileModel = new UserDocument;
if($callback) {
$filePath = $callback->storeAs('uploads', $fileName, 'public');
$fileModel->name = $fileName;
$fileModel->file_path = '/storage/' . $filePath;
$fileModel->save();
return back()
->with('success','File has been uploaded.')
->with('file', $fileName);
}
It saves a row to the db, albeit incorrectly, but it doesn't save the file to storage. I've reworked the $filePath line a million times, but I keep getting this error Call to a member function storeAs() on resource or something similar. I'm relatively new to working with Laravel, so I'm not sure what I should be looking for. Thoughts?
Removed everything and started over... got it! And for anyone else running into the same issue: just calling for a file that doesn't exist creates the file (unless the file exists - then it updates it), so you don't have to create a temp file or use $file = fopen('php://output', 'w'); to create the file. It'll automatically "save" the newly generated file in the file path you specified when you fclose() out of the file.
The only thing I'll note is that the file path has to exist (the file doesn't, but the file path does). In my instance, the file path already exists, but if yours doesn't or if you're not sure if it does, check to see if it exists, and then make the directory.
public function updatePaymentConfirm(Request $request) {
$user = Auth::user();
$path = storage_path('app/public/docs/user_docs/'.$user->id);
$fileName = $user->ein.'.csv';
$file = fopen($path.$fileName, 'w');
$columns = array('First Name', 'Email Address');
fputcsv($file, $columns);
$data = [
'First Name' => $user->first_name,
'Email Address' => $user->email,
];
fputcsv($file, $data);
fclose($file);
$symlink = 'public/docs/user_docs/'.$user->id.'/';
$fileModel = new UserDocument;
$fileModel->name = 'csv';
$fileModel->file_path = $symlink.$fileName;
$fileModel->save();
return redirect()->route('completed');
}
** UPDATE **
Everything worked perfectly locally, and when I pushed this to production, I received this error 🙄:
fopen(https://..../12-3456789.csv): failed to open stream: HTTP wrapper does not support writeable connections.
I'm saving to an s3 bucket, and I had to rework the entire process. You can't create and/or write to a file in the directory. I had to create a temp file first. Here's where I landed:
$user = Auth::user();
$s3 = Storage::disk('s3');
$storage = Storage::disk('s3')->url('/');
$path = 'public/docs/user_docs/'.$user->id.'/';
$csvFile = tmpfile();
$csvPath = stream_get_meta_data($csvFile)['uri'];
$fd = fopen($csvPath, 'w');
$columns = array('First Name', 'Email Address');
$data = array(
'First Name' => $user->first_name,
'Email Address' => $user->email,
);
fputcsv($fd, $columns);
fputcsv($fd, $data);
fclose($fd);
$s3->putFileAs('', $csvPath, $path.$user->ein.'.csv');
Today I have fixed it with this snipe:
// output up to 5MB is kept in memory, if it becomes bigger it will
// automatically be written to a temporary file
$csv = fopen('php://temp/maxmemory:'. (5*1024*1024), 'r+');
fputcsv($csv, array('blah','blah'));
rewind($csv);
$output = stream_get_contents($csv);
// Put the content directly in file into the disk
Storage::disk('myDisk')->put("report.csv", $output);
This code is easy and functional, use Laravel Storage Class
https://laravel.com/docs/9.x/filesystem#main-content
use Illuminate\Support\Facades\Storage;
// data array
$results = [
['id' => 0, 'name' => 'David', 'parent' => 1],
['id' => 1, 'name' => 'Ron', 'parent' => 0],
['id' => 2, 'name' => 'Mark', 'parent' => 1]
];
// create a variable to store data
$pages = "id,name,parent\n"; // use " not ' or \n not working
// use foreach to data
foreach ($results as $where) {
$pages .= "{$where['id']},{$where['name']},{$where['parent']}\n";
}
// use Fecades Laravel Storage
Storage::disk('local')->put('file.csv', $pages);
I have been making a plugin which allows admins to upload image, i can see the image(so can other people which is expected), delete and re-upload, but the problem is when another admin try to edit the plugin, they cant see the draft image, they only see the image they upload and i see the image i upload as draft. What i want is if i upload an image everyone that tries to edit, they should see the image i have upload as draft.
I will add my code below, if someone can help i will really appreciate it.
view.php:
$i = 1;
$out = array();
$fs = get_file_storage();
$files = $fs->get_area_files($this->context->id, 'block_cls_user_profile', 'cls_img', $i, false, false);
foreach ($files as $file) {
$filename = $file->get_filename();
$url = moodle_url::make_pluginfile_url(
$file->get_contextid(),
$file->get_component(),
$file->get_filearea(),
$file->get_itemid(),
$file->get_filepath(),
$filename);
// $out[] = html_writer::link($url, $filename);
}
// $br = html_writer::empty_tag('br');
lib.php
// Check the contextlevel is as expected - if your plugin is a block, this becomes CONTEXT_BLOCK, etc.
if ($context->contextlevel != CONTEXT_BLOCK) {
return false;
}
// Make sure the filearea is one of those used by the plugin.
if ($filearea !== 'cls_img') {
return false;
}
// $fs = get_file_storage();
// $file = $fs->get_file($context->id, 'local_myplugin', $filearea, $args[0], '/', $args[1]);
// send_stored_file($file);
// Make sure the user is logged in and has access to the module (plugins that are not course modules should leave out the 'cm' part).
require_login($course, true);
// No check for capability, because everybody needs to see it
// Check the relevant capabilities - these may vary depending on the filearea being accessed.
/*
if (!has_capability('mod/bookcase:addinstance', $context)) {
return false;
}
*/
// Leave this line out if you set the itemid to null in make_pluginfile_url (set $itemid to 0 instead).
$itemid = array_shift($args); // The first item in the $args array.
// Use the itemid to retrieve any relevant data records and perform any security checks to see if the
// user really does have access to the file in question.
// Extract the filename / filepath from the $args array.
$filename = array_pop($args); // The last item in the $args array.
if (!$args) {
$filepath = '/'; // $args is empty => the path is '/'
} else {
$filepath = '/'.implode('/', $args).'/'; // $args contains elements of the filepath
}
// Retrieve the file from the Files API.
$fs = get_file_storage();
$file = $fs->get_file($context->id, 'block_cls_user_profile', $filearea, $itemid, $filepath, $filename);
if (!$file) {
return false; // The file does not exist.
}
// We can now send the file back to the browser - in this case with a cache lifetime of 1 day and no filtering.
// From Moodle 2.3, use send_stored_file instead.
send_stored_file($file, 86400, 0, $forcedownload, $options);
edit_form.php
$mform->addElement('filemanager', 'config_profile_image', get_string('file'), null,
array('maxbytes' => $CFG->maxbytes, 'areamaxbytes' => 10485760, 'maxfiles' => 1,
'accepted_types' => '*', 'return_types'=> FILE_INTERNAL | FILE_EXTERNAL));
$i = 1;
if ($draftitemid = file_get_submitted_draft_itemid('config_profile_image')) {
file_save_draft_area_files($draftitemid, $this->block->context->id, 'block_cls_user_profile', 'cls_img', $i,array('subdirs' => false, 'maxfiles' => 1));
}
I have edited the edit_form.php and now it looks like this:
global $CFG, $mform;
$i = 0;
$draftitemid = file_get_submitted_draft_itemid('config_profile_image');
file_prepare_draft_area($draftitemid, $this->block->context->id, 'block_cls_user_profile', 'cls_img', $i,
array('subdirs' => 0, 'maxbytes' => $CFG->maxbytes, 'maxfiles' => 1));
$entry->config_profile_image = $draftitemid;
$mform->set_data($entry);
file_save_draft_area_files($draftitemid, $this->block->context->id, 'block_cls_user_profile', 'cls_img', $i,array('subdirs' => false, 'maxfiles' => 1));
parent::set_data($defaults);}}
But the error i get now is 'Call to a member function set_data() on null'.
Thank you.
foreach($new_files as $new_file) {
//create file
$myfile = fopen($filename, "w");
//put contents in the file
fwrite($filename, $new_file['content']);
//close the file
fclose($myfile);
}
I have this code to create a new file, I want to be able to open $filename, and create multiple files from it, with the content of new_file.
This code doesn't seem to work,all I get is one new empty file, any ideas?
Here is a working example.
You have to store data in standard structure like JSON in $filename, then read its data and make other files.
<?php
// $Data = file_get_contents($filename);
// $Data = json_decode($Data); <--- if stored as json
// Sample Data
$Data = [
[
'name' => 'file1.txt',
'content' => 'content1',
],
[
'name' => 'file2.txt',
'content' => 'content2',
],
];
foreach ($Data as $Row) {
$File = fopen($Row['name'], 'w');
fwrite($File, $Row['content']);
fclose($File);
}
Im trying to get a image from a PUT request for update a user picture(using postman), and make it pass through a validation in Laravel 5.2, for making the call in postman use the following url:
http://localhost:8000/api/v1/users?_method=PUT
and send the image string in the body, using a json like this:
{
"picture" : "data:image/png;base64,this-is-the-base64-encode-string"
}
In the controller try a lot of differents ways for decode the image and try to pass the validation:
First I tried this:
$data = request->input('picture');
$data = str_replace('data:image/png;base64,', '', $data);
$data = str_replace(' ', '+', $data);
$image = base64_decode($data);
$file = app_path() . uniqid() . '.png';
$success = file_put_contents($file, $image);
Then I tried this:
list($type, $data) = explode(';', $data);
list(, $data) = explode(',', $data);
$data = base64_decode($data);
$typeFile = explode(':', $type);
$extension = explode('/', $typeFile[1]);
$ext = $extension[1];
Storage::put(public_path() . '/prueba.' . $ext, $data);
$contents = Storage::get(base_path() . '/public/prueba.png');
Try to use the intervention image library (http://image.intervention.io/) and don't pass:
$image = Image::make($data);
$image->save(app_path() . 'test2.png');
$image = Image::make(app_path() . 'test1.png');
This is the validation in the controller:
$data = [
'picture' => $image,
'first_name' => $request->input('first_name'),
'last_name' => $request->input('last_name')
];
$validator = Validator::make($data, User::rulesForUpdate());
if ($validator->fails()) {
return $this->respondFailedParametersValidation('Parameters failed validation for a user picture');
}
this is the validation in the User-model:
public static function rulesForUpdate() {
return [
'first_name' => 'max:255',
'last_name' => 'max:255',
'picture' => 'image|max:5000|mimes:jpeg,png'
];
}
If you are using Intervention anyways, then you can leverage it for a custom validation rule. I have one called "imageable". It basically makes sure that the input given will be able to be converted to an Intervention image. Base64 data image strings will pass. A string of "foo" will not.
Validator::extend('imageable', function ($attribute, $value, $params, $validator) {
try {
ImageManagerStatic::make($value);
return true;
} catch (\Exception $e) {
return false;
}
});
This obviously just checks that the input is able to be converted to an image. However, it should show you as a use-case how you can leverage Intervention and any of it's methods to create a custom rule.
You can extend the Validator class of Laravel.
Laravel Doc
But anyway try this
Validator::extend('is_png',function($attribute, $value, $params, $validator) {
$image = base64_decode($value);
$f = finfo_open();
$result = finfo_buffer($f, $image, FILEINFO_MIME_TYPE);
return $result == 'image/png';
});
Don't forget the rules:
$rules = array(
'image' => 'is_png'
);
inside extend function add this
$res= mime_content_type($value);
if ($res == 'image/png' || $res == 'image/jpeg') {
return $res;
}
Hello: I have a web form that submits data to my db. I am able to create a signature image and save it within my directory. I want to save/store that signature image in my mysql db along with the record so I can call it later.
Written in CodeIgniter 2.0
here are my model and controller.
public function sig_to_img()
{
if($_POST){
require_once APPPATH.'signature-to-image.php';
$json = $_POST['output'];
$img = sigJsonToImage($json);
imagepng($img, 'signature.png');
imagedestroy($img);
$form = $this->input->post();
var_dump($form);
$this->coapp_mdl->insert_signature($form);
}
}
public function insert_signature($data)
{
$sig_hash = sha1($data['output']);
$created = time();
$ip = $_SERVER['REMOTE_ADDR'];
$data = array(
'first_name' => $data['fname'],
'last_name' => $data['lname'],
'signator' => $data['name'],
'signature' => $data['output'],
'sig_hash' => $sig_hash,
'ip' => $ip,
'created' => $created
);
return $this->db->insert('signatures', $data);
}
I found the function below on php.net but apparently I am doing something wrong or various things wrong. Does anyone know how to accomplish this functionality?
$imagefile = "changethistogourimage.gif";
$image = imagecreatefromgif($imagefile);
ob_start();
imagepng($image);
$imagevariable = ob_get_contents();
ob_end_clean();
Got it - For those curious here are the changes to my controller:
public function sig_to_img()
{
if($_POST){
require_once APPPATH.'signature-to-image.php';
$json = $_POST['output'];
$img = sigJsonToImage($json);
// Save to file
$file_name = trim(str_replace(" ","_",$_POST['name']));//name to used for filename
imagepng($img, APPPATH."../images/signatures/".$file_name.'.png');
$sig_name = $file_name.'.png'; //pass to model
// Destroy the image in memory when complete
imagedestroy($img);
$form = $this->input->post();
var_dump($form);
$this->coapp_mdl->insert_signature($form, $sig_name);
}
}