Column matching .csv to database using Symfony - php
I want to import a .csv file with headers such as name, email etc. as part of mail merge functionality in our web app (we are using Symfony 3.4). I have so far successfully managed to persist the .csv file to the database using Doctrine when I know what the headers on the .csv are going to be. I also need to be able to give the user the option of changing the header of the columns in case the headers are different or auto-matching on import doesn't work.
Is the best option here to persist the information to the database and then move the data using Doctrine, or is there some way to give the user the option to match columns before persisting? I am relatively new to both Symfony and Doctrine so I can't see the solution.
Here is the code from the controller:
public function usersListUploadAction(Request $request) {
// grab uploaded .csv file
$file = $_FILES["file"]["tmp_name"];
$fileimplode = implode("/", $file);
// convert .csv if made on mac
if (!ini_get("auto_detect_line_endings")) {
ini_set("auto_detect_line_endings", '1');
}
// create associative array from .csv
$reader = Reader::createFromPath($fileimplode);
$addedUsers = $reader->fetchAssoc();
$em = $this->getDoctrine()->getManager();
foreach($addedUsers as $row) {
// check for existing users
$emailAddress = $row["Email Address"];
$user = $this->get('manager_user.repository')->findByEmailAddressWithoutParents($emailAddress);
if ($user === []) {
// create new user
$userTypeId = 1;
$role = $request->get('roles', []);
$firstname = $row["First Name"];
$surname = $row["Last Name"];
$email = $row["Email Address"];
$username = (strtolower($row["First Name"].".".$row["Last Name"]));
$phone = null;
$userBuilder = $this->get('manager_user.manager')->newUser(
$userTypeId,
$username,
$phone,
$email,
$role,
$password = null,
$organisation = null,
$emailResetPasswordLink = 0,
$firstname,
$surname,
$name = null,
$roles = []
);
$user = $userBuilder;
// add email to contact_methods
$cmType = $this->get('manager_user_contact_method_type.repository')->find(ContactMethodTypeInterface::EMAIL);
$cmEmail = $this->get('manager_user_contact_method.builder')->createNew($cmType, $row["Email Address"], $user, 1);
$em->persist($userBuilder);
$em->persist($cmEmail);
}
$em->flush();
}
// return to user list page with list updated from .csv users
$listId = $request->get('listId');
$list = $this->get('manager.email.list_repository')->find($listId);
$status = 1;
//loop over the added users
foreach($addedUsers as $row) {
$userNameLower = (strtolower($row["First Name"].".".$row["Last Name"]));
$userId = strval($this->get('manager_user.repository')->getUserIdByUserName($userNameLower));
$user = $this->get('manager_user.repository')->find($userId);
if(!$user->isInList($list)) { //this user is not in the list already so add it
$dateSubscribed = new \DateTime();
$emailCampaignListUser = new EmailCampaignListUser($user, $list, $dateSubscribed, $status);
$em->persist($emailCampaignListUser);
}
}
$em->flush();
return $this->redirectToRoute('manager_email_campaign_lists_users_upload_check', array('listId' => 1));
}
I provide you an example (not full).
AS i ve commented you can process like this.
First step, the user can match expected fields with csv's header fields
Import data using the mapping previously made.
For the first step, you can choose the frontend solution you want, like Drag&Drop.
In my opinion you've to define your keys
//this is your code
//Here we define that "firstname" is your key for the first name
$fields = ["firstname","lastname"];
Second part of the first step is getting CSV header, using fgetcsv on the first line (header line)
if (($handle = fopen("yourcsv.csv", "r")) !== FALSE) {
//Here you get all csv headers
$header = fgetcsv($handle, 1000, ",");
fclose($handle);
}
At this point after you (or the user) do the mapping, you can store the mapping in database if this is a real need or simply use POST to get the mapping at the second step.
Second Step we parse the csv file skipping the first line and use the mapping previously done to get data correctly.
//Your mapping is stored in $mapping by example
//it contains "fieldname" => "csv index" example "firstname" => 3
if (($handle = fopen("test.csv", "r")) !== FALSE) {
//skip header line here
fgetcsv($handle, 1000, ",");
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
//$data contain each user data
$userBuilder = $this->get('manager_user.manager')->newUser(
$data[$mapping['firstname']],
$data[$mapping['lastname']],
//etc...
);
}
fclose($handle);
}
Hope this helps and clear.
Related
Encode-Decode issue php (CSV->MySQL)
I'm trying to pull data in CSV format for a server, process and store in a MySQL database: When I insert into my database, the text is "Atila%27s+Goonies" I want it to be "Atila's Goonies" I have been reading through all the text encoding topics, but I think I'm missing a very simple function. My requirement is to have this text and store in the database as a properly formatted (or encoded) string, so I can query later and display online. Database column is collated as "utf8_unicode_ci". I'm not sure if it's relevant because if I want to display on the browser, I cannot get to the correct format either. thanks. Any help would be appreciated. My code below: if (($handle = fopen("alliances.txt", "r")) !== FALSE) { while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) { $num = count($data); $row++; $alliance_id = $data[0]; $alliance_name = $data[1]; $alliance_points = $data[2]; $alliance_villages = $data[3]; $alliance_members = $data[4]; $alliance_rank = $data[5]; $sql_ia = "insert into alliances (alliance_id,alliance_name, alliance_villages,alliance_members,alliance_rank,timestamp) values (:alliance_id,:alliance_name, :alliance_villages,:alliance_members,:alliance_rank,:timestamp)"; $st_ia = $DBcon->prepare($sql_ia); $st_ia->bindParam(':alliance_rank', $alliance_rank,PDO::PARAM_INT); $st_ia->bindParam(':alliance_id', $alliance_id,PDO::PARAM_INT); $st_ia->bindParam(':alliance_name',$alliance_name,PDO::PARAM_STR); $st_ia->bindParam(':alliance_members',$alliance_members,PDO::PARAM_INT); $st_ia->bindParam(':alliance_villages',$alliance_villages,PDO::PARAM_INT); $st_ia->bindParam(':timestamp',$timestamp,PDO::PARAM_INT); $author_name = $ag_row[$agency]; $st_ia->execute(); } fclose($handle); }
Try this. <?php $encoded = 'Atila%27s+Goonies'; // Your string is URL encoded. Use urldecode() to decode. $decoded = urldecode($encoded); var_dump($decoded); // string(15) "Atila's Goonies"
Unable to insert data in CSV file PHP
I am in process of inserting data in the desired CSV file from another CSV file. CSV file is creating fine with out any problem but its is not insert array data in file. It only inserts header on the first row. Below is code I am trying: date_default_timezone_set('America/New_York'); set_time_limit(0); ini_set("memory_limit", -1); $realPath = realpath( dirname(__FILE__) ); $path = $realPath.'/3pltracking/'; $files = scandir($path); $FilePath = $path.$files[2]; $result = array(); $date = date('m-d-Y_his'); if (file_exists($FilePath)) { if (($handle = fopen($FilePath, "r")) !== FALSE) { $i=0; while (($data = fgetcsv($handle, 10000, ",")) !== FALSE) { $i++; if($i==1) continue; //$list = array('$data[2],$data[25],$data[4],$data[30],$data[41],$data[27]'); echo $data[2].",".$data[25].",".$data[4].",".$data[30].",".$data[41].",".$data[27]; echo "<br>"; $list = array($data[2].",".$data[25].",".$data[4].",".$data[30].",".$data[41].",".$data[27]); // the problem is here I believe as it is empty array if I check it outside while loop } fclose($handle); $headers = array('ReferenceNumber', 'TotalCartons', 'ShipCarrier', 'TrackingNum', 'FreightPP', 'TotalWeight'); $fp = fopen($realPath.'\3pltracking\TrackingFiles\Tracking_File_'.$date.'.csv', 'w'); fputcsv($fp, $headers); foreach ($list as $line) { $val = explode(",", $line); fputcsv($fp, $val); } fclose($fp); } else { $body = "File Not Found"; } } Here is my CSV file data: TransactionNumber,CustomerName,ReferenceNumber,PurchaseOrderNumber,ShipCarrier,ShipService,ShipBilling,ShipAccount,EarliestShipDate,CancelDate,Notes,ShipToName,ShipToCompany,ShipToAddress1,ShipToAddress2,ShipToCity,ShipToState,ShipToZip,ShipToCountry,ShipToPhone,ShipToFax,ShipToEmail,ShipToCustomerName,ShipToDeptNumber,ShipToVendorID,TotalCartons,TotalPallets,TotalWeight,TotalVolume,BOLNum,TrackingNum,TrailerNum,SealNum,ShipDate,ItemNumber,ItemQuantityOrdered,ItemQuantityShipped,ItemLength,ItemWidth,ItemHeight,ItemWeight,FreightPP,WarehouseID,LotNumber,SerialNumber,ExpirationDate,Supplier,Cost,FulfillInvShippingAndHandling,FulfillInvTax,FulfillInvDiscountCode,FulfillInvDiscountAmount,FulfillInvGiftMessage,SoldToName,SoldToCompany,SoldToAddress1,SoldToAddress2,SoldToCity,SoldToState,SoldToZip,SoldToCountry,SoldToPhone,SoldToFax,SoldToEmail,SoldToCustomerID,SoldToDeptNumber,FulfillInvSalePrice,FulfillInvDiscountPct,FulfillInvDiscountAmt 242328,PARADIGM TRENDS,123810,40-402849,CUSTOMER PICK UP,LTL,FreightCollect,,,,,HG BUYING- JEFFERSON DC 884,HG BUYING- JEFFERSON DC 884,125 LOGISTICS CENTER PKWY,,JEFFERSON,AL,30549,US,,,,,,,30,0,30,0.0174,,,,,,DOV3S,64,64,4,1,1,4,0,1,,,,,,0,0,,0,,,,,,,,,,,,,,,0,0,0 33,d,123810,40-402849,CUSTOMER PICK UP,LTL,FreightCollect,,,,,HG BUYING- JEFFERSON DC 884,HG BUYING- JEFFERSON DC 884,125 LOGISTICS CENTER PKWY,,JEFFERSON,AL,30549,US,,,,,,,30,0,30,0.0174,,,,,,DOV3S,64,64,4,1,1,4,0,1,,,,,,0,0,,0,,,,,,,,,,,,,,,0,0,0 44,PARAdgdfDIGM TRENDS,123810,40-402849,CUSTOMER PICK UP,LTL,FreightCollect,,,,,HG BUYING- JEFFERSON DC 884,HG BUYING- JEFFERSON DC 884,125 LOGISTICS CENTER PKWY,,JEFFERSON,AL,30549,US,,,,,,,30,0,30,0.0174,,,,,,DOV3S,64,64,4,1,1,4,0,1,,,,,,0,0,,0,,,,,,,,,,,,,,,0,0,0 ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,BY3M,176,176,11,1,1,11,,,,,,,,,,,,,,,,,,,,,,,,,,0,0,0
There are so many ways of going about this... including str_getcsv($csvData). However here we'd go for something old-school & a bit twisted;-). We would create a Function that uses Regex and a Looping Construct to build-up the relevant CSV Data Structure. The Function below illustrates how. Also note that although we mentioned that this is a somewhat twisted, old-school approach: don't be fooled... because it does its thing still ;-). <?php $csvSourceFile = __DIR__ . "/1.csv"; $csvPreferredColumns = array('ReferenceNumber', 'TotalCartons', 'ShipCarrier', 'TrackingNum', 'FreightPP', 'TotalWeight'); $newCsvStrut = processCSVData($csvSourceFile, $csvPreferredColumns, __DIR__ . "/test.csv"); /** * #param $csvSource // PATH TO THE MAIN CSV FILE * #param array $csvPreferredColumns // ARRAY OF HEADER COLUMN-NAMES TO BE EXTRACTED FROM MAIN CSV * #param null $newCSVFileName // NAME OF THE NEW CSV FILE TO BE CREATED. * #return string */ function processCSVData($csvSource, array $csvPreferredColumns, $newCSVFileName=null){ // GET THE CONTENTS OF THE CSV FILE & STORE IT IN A VARIABLE $csvData = file_get_contents($csvSource); // SPLIT THE CONTENTS OF THE CSV FILE LINE BY LINE: THAT IS; AT THE END OF EACH LINE // THUS CONVERTING THE DATA TO AN ARRAY... $arrCsvLines = preg_split("#\n#", $csvData); //FILTER OUT UNWANTED EMPTY VALUES FROM THE ARRAY $arrCsvLines = array_filter($arrCsvLines); // CREATE SOME VARIABLES TO BE USED WITHIN THE LOOP... $strDataFinal = ""; $arrDataMain = $arrDataFinal = array(); // IF THERE IS MORE THAN ONE LINE IN THE ARRAY WE CREATED ABOVE, // THEN CONTINUE PROCESSING THE DATA... if($arrCsvLines && count($arrCsvLines)>0){ // SINCE THE HEADER IS ALWAYS THE FIRST LINE IN THE CHAIN, // WE EXPLICITLY EXTRACT IT AND STORE IT IN A VARIABLE FOR LATER USE $arrCsvHeaders = preg_split("#\,([\s\t]+)?#", $arrCsvLines[0]); // NOW WE LOOP THROUGH ALL THE LINES WE CREATED BY SPLITTING THE CONTENTS // OF THE CSV FILE AT THE END-OF-LINE BOUNDARY foreach($arrCsvLines as $key=>$arrCsvLine){ // WE DON'T WANT ANYTHING AT INDEX "0" SINCE IT IS THE HEADER // AND WE ALREADY DEALT WITH IT ABOVE.... // SO IF THE INDEX $key IS NOT 0, WE CAN CONTINUE PROCESSING if($key != 0){ $arrDataTemp = array(); $arrTempCsvData = preg_split("#\,([\s\t]+)?#", $arrCsvLine); foreach($arrTempCsvData as $iKey=>$sData){ $arrDataTemp[$arrCsvHeaders[$iKey]] = $sData; } $arrDataMain[] = $arrDataTemp; } } foreach($arrDataMain as $iKey=>$subData){ $arrTempFinal = array(); foreach($subData as $key=>$data){ if(in_array($key, $csvPreferredColumns)){ $arrTempFinal[$key] = $data; } } $arrDataFinal[] = implode(",\t", $arrTempFinal); } $arrDataFinal = array_merge( array(implode(",\t", $csvPreferredColumns)), $arrDataFinal); $strDataFinal = implode("\n", $arrDataFinal); if($newCSVFileName){ file_put_contents($newCSVFileName, $strDataFinal); } } return $strDataFinal; } var_dump($newCsvStrut); // PRODUCES SOMETHING SIMILAR TO THE LINES BELOW: string 'ReferenceNumber, TotalCartons, ShipCarrier, TrackingNum, FreightPP, TotalWeight 123810, CUSTOMER PICK UP, 30, 30, , 0 123810, CUSTOMER PICK UP, 30, 30, , 0 123810, CUSTOMER PICK UP, 30, 30, , 0 , , , , , ' (length=204)
Turning indexed array to associative array that still works with a CSV file. -PHP
I can't seem to find an answer. Is there an easy way to change my array in the code, to an associative array? NOTE: -All the variables being assigned are declared already -It basically creates an array from a csv file called staff data and loops through to find the next empty line then adds that row with data, then re-saves back into the csv file. My problem is that I cannot figure out in the slightest how to change this array to an associative array that will still work with my code.. Any direction would be greatly appreciated. I hope my problem is clear. // Grabs the csv file (and its existing data) and makes it into an array so the new data can be added to it. $StaffDetails = array(); $lines = file('data/StaffData.csv', FILE_IGNORE_NEW_LINES); foreach ($lines as $key => $value) { $StaffDetails[$key] = str_getcsv($value); } //This is a for loop that checks when there is an avaliable line to put the data into. It loops over the lines, until there is an empty spot for the new data //to go into. //It will not run it if the full name is empty. for ($i=0; $i<200; $i++) { if(empty($StaffDetails[$i][0]) == false){ //If its full }else if (empty($StaffDetails[$i][0]) == true){ //If it is empty //This prints the details of this staff to that line in the data file. $i is the avaliable line found above. $StaffDetails[$i][0] = $UserFullName; $StaffDetails[$i][1] = $UserPhoneNumber; $StaffDetails[$i][2] = $Gender; $StaffDetails[$i][3] = $Birthday; $StaffDetails[$i][4] = $TypeOfWork; $StaffDetails[$i][5] = $StartingDate; $StaffDetails[$i][6] = $AnnualLeaveLeft; $StaffDetails[$i][7] = $SickLeaveLeft; $StaffDetails[$i][8] = $ID; $StaffDetails[$i][9] = $NumberOfWorkDaysAWeek; $StaffDetails[$i][10] = $Monday; $StaffDetails[$i][11] = $Tuesday; $StaffDetails[$i][12] = $Wednesday; $StaffDetails[$i][13] = $Thursday; $StaffDetails[$i][14] = $Friday; $StaffDetails[$i][15] = $Saturday; $StaffDetails[$i][16] = $Sunday; $StaffDetails[$i][17] = $UserUsername; $StaffDetails[$i][18] = $UserPassword; $StaffDetails[$i][19] = $Em1FullName; $StaffDetails[$i][20] = $Em1PhoneNumber; $StaffDetails[$i][21] = $Em2FullName; $StaffDetails[$i][22] = $Em2PhoneNumber; $i =201; } //Below saves the previous data and new changes to the csv file //This opens the csv file as a write file $MyCsvFile = fopen('data/StaffData.csv', 'w'); //This takes the array that has had data ddded to it and adds it to the file foreach ($StaffDetails as $fields) { fputcsv($MyCsvFile, $fields); } //This closes the file fclose($MyCsvFile); }
Updating CSV Column Values Using UA-Parser Library
I'm using the ua-parser library to identify the device family for a number of user agent strings in a spreadsheet column. The problem I'm running into is that it doesn't seem like my function is really running. The value output for detectAgent($data[2]) is not always accurate. Here's a code sample. I feel like I must be missing something related to the limitations of creating objects over and over again. Thanks in advance for any help. <?php require_once 'vendor/autoload.php'; use UAParser\Parser; function detectAgent($ua) { $parser = Parser::create(); $result = $parser->parse($ua); return $result->os->family; } $input_file = "input.csv"; $output_file = "output.csv"; if (($handle1 = fopen($input_file, "r")) !== FALSE) { if (($handle2 = fopen($output_file, "w")) !== FALSE) { while (($data = fgetcsv($handle1, 5000000, ",")) !== FALSE) { // Alter your data #print $data . "<br />"; $data[2] = detectAgent($data[2]); //identify browser family // Write back to CSV format fputcsv($handle2, $data); } fclose($handle2); } fclose($handle1); } ?>
This was a silly mistake. I was writing to the wrong column in $data[2] = detectAgent($data[2]);. If anyone else runs into the same problem, the code is working now and I've posted an example here.
Parse large CSV into mysql db
I want to parse very large csv into mysql using php. My idea is to make it with executing 4-5 times the request which will paste data into mysql, after each iteration request should start from rows which has not been pasted yet (with one request iteration script can cover 400 rows). But I don't know to skip rows in csv, which has been already pasted into database. Maybe I should check number of rows in the table then define some variable with this number and make iteration according on it. But I don't know how to operate with csv using foreach, I only have code with "while" from example. Here is my current code: public function action_index(){ if(($handle = fopen('data_wpic.csv', 'r')) !== false) { $header = fgetcsv($handle); while(($data = fgetcsv($handle)) !== false) { $model = ORM::factory('Drug'); $image_path = $data[18]; if(strlen($image_path) > 5) { $path= 'drug_images/' . $image_path; $image = ORM::factory('Image')->remote($path); if ($image) { $model->image_id = $image; unlink($path); } } $model->drugGenericName = $data[17]; $model->drugForm = $data[4]; $model->drugProperties = $data[7]; $model->drugIndication = $data[2]; $model->drugDosage = $data[13]; $model->drugSide = $data[11]; $model->drugContrIndication = $data[12]; $model->drugInteractions = $data[15]; $model->drugSpecial = $data[0]; $model->drugExpiry = $data[3]; $model->drugRealCondition = $data[8]; $model->tradeName = $data[16]; $model->save(); unset($data); } fclose($handle); } } I have limited web hosting, thats why I try to solve this routine If someone knows better aproach - I will be glad to hear him