I'm trying to read a list of items from a CSV file, compare it with the items in my database, and generate a newly one with the ones not in my base. From the CSV with thousand results, only 26 were not in db. However, the first item in my new CSV is present in my database, meaning it's a false positive. Only the first item is wrong, all the others are fine (I've queried them all).
Here is my code:
<?php
function generate_diff_csv() {
$conn = new mysqli("localhost","rcpp","*********", "items");
$key_ref = fopen("INV14.csv", "r");
$not_in = fopen("not_in.csv","w");
[...]
fclose($key_ref);
$keys = array();
foreach ($custom1 as $custom) {
$trimmed_custom = trim($custom);
$result = $conn->query("SELECT custom1 FROM products WHERE custom1 = '{$trimmed_custom}'");
if($result->num_rows == 0) {
$keys[] = array("key" => $trimmed_custom);
echo "adicionado ao csv... \n";
}
}
foreach($keys as $key) {
fputcsv($not_in, $key);
}
fclose($not_in);
$conn->close();
}
generate_diff_csv();
To be sure I had everything right, I created a temporary table with the data I needed to compare. When I query it with an SQL, I get the 25 results. Putting them (PHP x SQL) side-by-side in a file, only the first is not a match, meaning it is really the only wrong result.
SELECT ref FROM refs WHERE ref NOT IN (SELECT custom1 FROM products);
Why is that? Why PHP returns the 1st key on my query?
The PHP is being executed from the command line, PHP 5.4.12 (Windows). I haven't tested on the Linux production environment, but I don't believe this would be a platform specific issue.
Thank you in advance.
Well, first of all, depending on the collation in the database, table and field, SQL Might be ignoring letter case, so for SQL, "something" is identical to "SoMEthIng". Also, since my practice with SQL CHAR() value types, it has come to my attention that suffix spaces are also ignored in comparison, at least when it comes to CHAR().
I've solved the problem. And it was a platform problem, but not related with PHP (I think).
I was running the script through cmd, not Powershell, and the first item was getting an additional UTF BOM header on the first character. My fault was to not pay attention to the first output, thinking it was just cmd pritting some rubbish characters. But when I used var_dump($custom1), I could see that those characters were being put inside the variable and trim wasn't cleaning it.
The solution was to remove BOM from CSV input file.
Link reference (for the BOM character issue): https://superuser.com/questions/601282/is-not-recognized-as-an-internal-or-external-command --- first answer explains why 'cmd' outputs those characters.
Related
So, It is very odd that with certain files this program just breaks. I have input files with 500 rows on it, and it works just fine, but if I try to input 1000 lines or more, the program only gets the first row of the csv file (where the titles are) and breaks the whole while loop.
I noticed there were another questions that looked similar; however, when I read through it I realized they weren't calling fgetcsv() into a loop. In contrast, I am calling the function in a while loop.
My code looks like this:
if(move_uploaded_file($_FILES["fileToUpload"]["tmp_name"], $target_file)){
$goe = fopen($target_file, "r");
while($data = fgetcsv($goe, filesize($target_file))){
if(!empty($data[0]) && !empty($data[2])){
if($data[0] !='brand' && $data[2] !='MPN'){
$string = $data[0] .' '. $data[2];
$arrayOfSearches[$data[1]] = $string;
}
}
}
fclose($target_file);
}
After debugging I realized it enter once in the while loop, it enters into the first if condition, but when it gets to the second one it never enters (makes sense since the first row are the titles which I don't want).
Any ideas?
I recently had the same problem, when reading a cvs file, the process was returning all rows in one line, but luckily I found the solution, just add this line to your PHP file:
ini_set('auto_detect_line_endings', true);
Hope works for you.
(Can't paste the exact question as the contest is over and I am unable to access the question. Sorry.)
Hello, recently I took part in a programming contest (PHP). I tested the code on my PC and got the desired output but when I checked my code on the contest website and ideone, I got wrong output. This is the 2nd time the same thing has happened. Same PHP code but different output.
It is taking input from command line. The purpose is to bring substrings that contact the characters 'A','B','C','a','b','c'.
For example: Consider the string 'AaBbCc' as CLI input.
Substrings: A,a,B,b,C,c,Aa,AaB,AaBb,AaBbC,AaBbCc,aB,aBb,aBbC,aBbCc,Bb,BbC,BbCc,bC,bCc,Cc.
Total substrings: 21 which is the correct output.
My machine:
Windows 7 64 Bit
PHP 5.3.13 (Wamp Server)
Following is the code:
<?php
$stdin = fopen('php://stdin', 'r');
while(true) {
$t = fread($stdin,3);
$t = trim($t);
$t = (int)$t;
while($t--) {
$sLen=0;
$subStringsNum=0;
$searchString="";
$searchString = fread($stdin,20);
$sLen=strlen($searchString);
$sLen=strlen(trim($searchString));
for($i=0;$i<$sLen;$i++) {
for($j=$i;$j<$sLen;$j++) {
if(preg_match("/^[A-C]+$/i",substr($searchString,$i,$sLen-$j))) {$subStringsNum++;}
}
}
echo $subStringsNum."\n";
}
die;
}
?>
Input:
2
AaBbCc
XxYyZz
Correct Output (My PC):
21
0
Ideone/Contest Website Output:
20
0
You have to keep in mind that your code is also processing the newline symbols.
On Windows systems, newline is composed by two characters, which escaped representation is \r\n.
On UNIX systems including Linux, only \n is used, and on MAC they use \r instead.
Since you are relying on the standard output, it will be susceptible to those architecture differences, and even if it was a file you are enforcing the architecture standard by using the flag "r" when creating the file handle instead of "rb", explicitly declaring you don't want to read the file in binary safe mode.
You can see in in this Ideone.com version of your code how the PHP script there will give the expected output when you enforce the newline symbols used by your home system, while in this other version using UNIX newlines it gives the "wrong" output.
I suppose you should be using fgets() to read each string separetely instead of fread() and then trim() them to remove those characters before processing.
I tried to analyse this code and that's what I know:
It seems there are no problems with input strings. If there were any it would be impossible to return result 20
I don't see any problem with loops, I usually use pre-incrementation but it shouldn't affect result at all
There are only 2 possibilities for me that cause unexpected result:
One of the loops iteration isn't executed - it could be only the last one inner loop (when $i == 5 and then $j == 5 because this loop is run just once) so it will match difference between 21 and 20.
preg_match won't match this string in one of occurrences (there are 21 checks of preg_match and one of them - possible the last one doesn't match).
If I had to choose I would go for the 1st possible cause. If I were you I would contact concepts author and ask them about version and possibility to test other codes. In this case the most important is how many times preg_match() is launched at all - 20 or 21 (using simple echo or extra counter would tell us that) and what are the strings that preg_match() checks. Only this way you can find out why this code doesn't work in my opinion.
It would be nice if you could put here any info when you find out something more.
PS. Of course I also get result 21 so it's hard to say what could be wrong
I'm creating a C# to PHP Data Connector to allow for a standardized connection to a web server to host data from a database to a C# WinForm application. Everything is working with this one small exception.
The basic of use is this.
C# sends an AES encrypted command to the server. The server parses the command and performs the SQL query and returns an AES encrypted string. This string is then converted to a DataTable in C#.
When the SQL contains a column that is a BLOB I'm only getting back a small part of the full data. It seems that the field is being limited to only the first 2792 bytes.
Is there a setting that is preventing the full contents of the BLOB to be returned?
I'm not sure if it will be helpful, but here is the code that does the work.
$DataConnection = new PDO('mysql:host=10.10.100.102;dbname=jmadata', "root", "nbtis01");
$DataConnection->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
if (isset($Parameters['SQLQuery'])) { // Default List
$SQLQuery = $Parameters['SQLQuery'];
unset($Parameters['SQLQuery']);
}
if (isset($Parameters['LimitOverride'])) {
if (!strpos(strtoupper($SQLQuery), "LIMIT"))
$SQLQuery = rtrim($SQLQuery, ';') . " LIMIT " . $Parameters['LimitOverride'];
unset($Parameters['LimitOverride']);
}
$QueryParams = array();
foreach ($Parameters as $key => $value)
if ($key !== '')
$QueryParams[$key] = $value;
$Query = $DataConnection->prepare($SQLQuery);
$Query->execute($QueryParams);
$ReturnArray = $Query->fetchAll(PDO::FETCH_ASSOC);
if (!$ReturnArray)
$ReturnArray[0] = array("NoResults" => "");
EDIT -- ANSWER
I found my issue. The problem had nothing to do with PDO, PHP or MySQL. I was taking the BLOB data and doing a Base64 to it before putting it in the array, as the split characters I was using to build the result string that would be converted to datatable in c# used non-printable characters and the binary data as a string might have included these characters. The issue was when I was doing a convert in c# to get the original string so that could convert that to a byte array. I was using System.Text.Encoding.ASCII.GetString to convert the Base64 byte array to the original string. This was working on everything but the binary data from the BLOB fields.
The suggestion that it might be a terminating character is what made me find it. Once the Base64 was converted to string using ASCII there was something that was turning into a terminator and it was stopping the convert at that point. Once I found this I changed to System.Text.Encoding.Default.GetString and now it works perfect.
Posted the answer in case anyone else might be trying to do this and having this same issue.
More details in the Edit of the question.
Changed from System.Text.Encoding.ASCII.GetString to System.Text.Encoding.Default.GetString and the issue was resolved.
Thank you crush for pointing me in the right direction to find the solution.
I have a log file (log.txt) in the form:
=========================================
March 01 2050 13:05:00 log v.2.6
General Option: [default] log_options.xml
=========================================
Loaded options from xml file: '/the/path/of/log_options.xml'
printPDF started
PDF export
PDF file created:'/path/of/file.1.pdf'
postProcessingDocument started
INDD file removed:'/path/of/file.1.indd'
Error opening document: '/path/of/some/filesomething.indd':Error: file doesnt exist or no permissions
=========================================
March 01 2050 14:15:00 log v.2.6
General Option: [default] log_options.xml
=========================================
Loaded options from xml file: '/the/path/of/log_options.xml'
extendedprintPDF started
extendedprintPDF: Error: Unsaved documents have no full name: line xyz
Note: Each file name is of the format: 3lettersdatesomename_LO.pdf/indd. Example: MNM011112ThisFile_LO.pdf. Also, on a given day and time, the entry could either have just errors, just the message about the file created or both, like I have shown here.
The file continues this way. And, I have a db in the form:
id itemName status
1 file NULL
And so on...
Now, I am expected to go through the log file and for each file that is created or if there in an error, I am supposed to update the last column of DB with appropriate message: File created or Error. I thought of searching the string "PDF file created/Error" and then grabbing the file name.
I have tried various things like pathinfo() and strpos. But, I can't seem to understand how I am going to get it done.
Can someone please provide me some inputs on how I can solve this? The txt file and db are pretty huge.
NOTE: I provided the 2nd entry of the log file to be clear that the format in which errors appear IS NOT consistent. I would like to know if I can still achieve what I am supposed to with an inconsistent format for errors.
Can somebody please help after reading the whole question again? There have been plenty of changes from the first time I posted this.
You can use the explode method of php to break your file into pieces of words.
In case the fields in your text file are tab separated then you can explode on explode(String,'\t'); or else in case of space separated, explode on space.
Then a simple substr(word,start_index,length) on each word can give you the name of file (here start_index should be 0).
Using mysql_connect will help you connect to mysql database, or a much efficient way would be to use PDO (PHP Data Objects) to make your code much more reliable and flexible.
Another way out would be to use the preg_match method and specify a regular expression matching your error msg and parse for the file name.
You can refer to php.net manual for help any time.
Are all of the files PDFs? If so you can do a regex search on files with the .pdf extension. However, if the filename is also contained in the error string, you will need to exclude that somehow.
// Assume filenames contain only upper/lowercase letters, 0-9, underscores, periods, dashes, and forward slashes
preg_match_all('/([a-zA-Z0-9_\.-/]+\.pdf)/', $log_file_contents, $matches);
// $matches should be an array containing each filename.
// You can do array_unique() to exclude duplicates.
Edit: Keep in mind, $matches will be a multi-dimensional array as described http://php.net/manual/en/function.preg-match-all.php and http://php.net/manual/en/function.preg-match.php
To test a regex expression, you can use http://regexpal.com/
Okay, so the main issue here is that you either don't have a consistent delimiter for "entries"..or else you are not providing enough info. So based on what you have provided, here is my suggestion. The main caveat here is that without a solid delimiter for "entries," there's no way to know for sure if the error matches up with the file name. The only way to fix this is to format your file better. Also you have to fill in some blanks, like your db info and how you actually perform the query.
$handle = fopen("log.txt", "rb");
while (!feof($handle)) {
// get the current row
$row = fread($handle, 8192);
// get file names
preg_match('~^PDF file created:(.*?)$~',$row,$match);
if ( isset($match[1]) ) {
$files[] = $match[1];
}
// get errors
preg_match('~^Error:(.*?)$~',$row,$match);
if ( isset($match[1]) ) {
$errors[] = $match[1];
}
}
fclose($handle);
// connect to db
foreach ($files as $k => $file) {
// assumes your table just has basename of file
$file = basename($file);
$error = ( isset($errors[$k]) ) ? $errors[$k] : null;
$sql = "update tablename set status='$error' where itemName='$file'";
// execute query
}
EDIT: Actually going back to your post, it looks like you want to update a table not insert, so you will want to change the query to be an update. And you may need to further work with $file in that foreach for your where clause, depending on how you store your filenames in your db (for example, if you just store the basename, you will likely want to do $file = basename($file); in the foreach). Code updated to reflect this.
So hopefully this will point you in the right direction.
I have records into .CSV file and I want to import them into MySQL database.
Whenever I import the .CSV I get the message Import has been successfully finished... but only 79 out of 114 records are be inserted into the database.
When I try to import the .CSV file with 411 records, just 282 are be inserted. The CSV file which got 411 records includes two categories of records Active and Sold whereby 114 records are Active.
Has someone gotten this type of problem? If so what should be done?
I wrote my own csv importer with php. I use php command fgetcsv to read the csv file and then I use mysql insert command in a loop.
$handle = fopen($this->file, "r");
$i=0;
$delimiter = ($this->fieldDelimiter == 'TAB') ? chr(9) : $this->fieldDelimiter;
while (($data = fgetcsv($handle, 10000, $delimiter)) !== FALSE)
{
$mydata[] = $data;
}
fclose ($handle);
reset ($mydata);
if ($this->CSVhasTitle)
{
$mydata = array_slice($mydata,1); //delete first row
}
Then I loop through my array and I use mysql insert:
foreach ($mydata as $value)
{
INSERT INTO $table (...) VALUES (....)
}
But I add exact columnnames into the array before the loop. I've an array of all columnames.
I had this problem too. Even though its a bit old and these recommendations go in one direction here is the solution I found. I was creating a large database and import and had the same thing happen, after trial and trials I realized that a key was created somehow assigned (I didn't recognize it because Iwas using the new skin and always use the old skin). Took out the key that somehow got assigned not by me and left the primary only and boom, absorbed the data upload. Also, had issues with view now, and request times out with respect to your question. I pushed up the viewer display a lot to thousands and now stuck, cannot access the config file anywhere with my CP. So as will hang and hosting customer support to lazy to read my concerns and override it on their end-I will have to remove whole table instead of any DROP as can't even run SQL as freezes with overload. So, food for thought would be to keep you table view down, which sucks like in my case b/c I need to look at 17,000 rcords visually quickly to ensure my .csv was correct rather then functions as if issues then can spot them and correct in the control which makes more sense to me anyway.
Take a look at your CSV file. It very likely contains something like
1,2,"some data",1
2,5,"data,with,comma",2
If you don't specify COLUMNS OPTIONALLY ENCLOSED BY '"' (SINGLE_QUOTE DOUBLE_QUOTE SINGLE_QUOTE) then the commas embedded in the string data in the second row, third column will not be imported properly.
Check the CSV to see what enclosure character is being used and specify that in the phpmyadmin interface.