I have a large file that I would like to read via php, and then insert various fields into MySQL.
Each file in the feed is in plain text format, separated into columns and rows. Each record has the same set of fields. The following are the delimiters for each field and record:
Field Separator (FS): SOH (ASCII character 1)
Record Separator (RS) : STX (ASCII character 2) + "\n"
If I look at the first few lines of the file they look like this:
#export_dateapplication_idlanguage_codetitledescriptionrelease_notescompany_urlsupport_urlscreenshot_url_1screenshot_url_2screenshot_url_3screenshot_url_4screenshot_width_height_1screenshot_width_height_2screenshot_width_height_3screenshot_width_height_4
#primaryKey:application_idlanguage_code
#dbTypes:BIGINTINTEGERVARCHAR(20)VARCHAR(1000)LONGTEXTLONGTEXTVARCHAR(1000)VARCHAR(1000)VARCHAR(1000)VARCHAR(1000)VARCHAR(1000)VARCHAR(1000)VARCHAR(20)VARCHAR(20)VARCHAR(20)VARCHAR(20)
#exportMode:FULL
I am struggling to no where to start in order to read this file into PHP, can anyone help with the basic PHP to read each record, and assign a variable to each field, which I then will be able to write into MySQL. I can handle the writing into SQL once I have the various fields set up.
Thanks in advance,
Greg
files greator than 2gb cant be read in PHP (32 bit limit).
For lower size use simple fopen function
And inserting mysql is all the work of macthing patterns and inserts.
If structure of table is same every row then better make it manual once and then just execute inserts by extracting values either by regex or other functions like explode and split .
If every line has delimiters between each field, you may look at fgetcsv().
When you use fgetcsv() on a line, it will return an array with the contents from that line. Since you have several lines, put the funciton inside a while()-loop (look at example #1)
Related
I want to update a Serialized value on meta_value column on mysql table.But json is hard to select from sql . How can i achieve.
a:2:{i:0;a:2:{s:19:"sevent_speaker_name";s:8:"John Doe";s:18:"sevent_speaker_img";a:1:{i:0;s:5:"72921";}}i:1;a:2:{s:19:"sevent_speaker_name";s:10:"John Smith";s:18:"sevent_speaker_img";a:1:{i:0;s:5:"72922";}}}
Here is the value. I want to replace all sevent with elevent but how can i do it?
Can I use LIKE? But first it need to be unserialized?.
This encoded information probably comes from WordPress or some other PHP framework's database; it's not JSON. Note that strings are encoded by storing the string length and the string contents:
s:19:"sevent_speaker_name"
You can use MySQL's REPLACE function to replace sevent with elevent, but you must be careful to update the length value s as well, or WordPress/PHP won't be able to read in the data.
It's possible to write a MySQL query to update the specific example given above, but it's difficult to write a query to substitutes all strings generically. Here is a tool that does the work.
I have a tab delimited text file with the first row being label headings that are also tab delimited, for example:
Name ID Money
Tom 239482 $2093984
Barry 293984 $92938
The only problem is that there are 30 some columns instead of 3 so I'd rather not have to type out all the (name VARCHAR(50),...) if it's avoidable.
How would I go about writing a function that creates the table from scratch in php from the text file, and say the function takes in $file_path and $table_name? Do I have to write all the column names again telling mysql what type they are and chop off the top or is there a more elegant solution when the names are already there?
You would somehow need to map the column type to the columns in your file. You could do this by adding that data to your textfile. For instance
Name|varchar(32) ID|int(8) Money|int(10)
Tom 239482 $2093984
Barry 293984 $92938
or something similar. Then write a function thet get's the column name and columntype using the first line and the data to fill the table with using all the other rows. You might also want to add a way to name the given table etc. However, this would probably be as much work (if not more) than creating SQL queries using you text file. Add a create table statement at the top and insert statements for each line. With search and replace this could be done very fast.
Even if you could find a way to do this, how would you determine the column type? I guess there would be some way to determine the type of the columns through checking for certain attributes (int, string, etc). And then you'd need to handle weird columns like Money, which might be seen as a string because of the dollar sign, but should almost certainly be stored as an integer.
Unless you plan on using this function quite a bit, I wouldn't bother spending time cobbling it together. Just fat finger the table creation. (Ctrl-C, Ctrl-V is your friend)
A group of people have been inconsistently entering data for a while.
Some people will enter this:
101mxeGte - TS 200-10
And other people will enter this
101mxeGte-TS-200-10
The sad thing is, those are supposed to be identical records.
They will also search inconsistently. If a record was entered one way, some people will search the other way.
Now, I know all about how you can fix data entry for the future, but that's NOT what I am asking about. I want to know how it is possible to:
Leave the data alone, but...
Search for the right thing.
Am I asking for the impossible here?
The best thing I found so far was a suggestion to simply muck about with the existing data, using the REPLACE function in mySQL.
I am uncomfortable with this option, as it means it will certainly actively piss off half of the users. The unfocused angst of all is less than the active ire of half.
The problem is that it has to go both ways:
Entering spaces in the query has to find both space and not-space entries,
and NOT entering spaces ALSO has to find both space and not-space entries.
Thanks for any help you can offer!
The "ideal" solution is pretty straightforward:
Decide what is the canonical way of representing a record
When someone saves a record, canonicalize it before saving
When someone searches for a record, canonicalize the input before searching for it
You could also write a small program to convert all existing data to the canonical form (you will have the code for it anyway, as "canonicalize" in steps 2 and 3 require that you write code that does so).
Edit: some specific information on how to canonicalize
With the sample data you give, the algorithm might be:
Replace all spaces with hyphens
Replace all runs of one or more hyphens with a single hyphen (a regex would be easiest for this -- actually, a regex can do both steps in one go)
Is there any practical problem with this approach?
Trim whitespaces from BOTH the existing data and the input of the search. That way the intended record(s) will always be returned. Hope your data size is small, though, because it's going to perform pretty poorly.
Edit: by "existing data" I meant "the query of existing data". My answer was based on assumption that the actual data could not be touched (which might not be correct).
If it where up to me, I'd have the data in the database updated with REPLACE, and on future searches when dealing with the given row remove all spaces in the input.
Presumably your users enter the search terms (or record details, when creating a record) in an HTML form, which then goes to a PHP script. It looks like your data can always be written in a way that contains no spaces, so why don't you do this:
Run a query that strips spaces from the existing data
Add code in the PHP script(s) that receives the form(s), so that it strips spaces from submitted data - whether that data is to be used for search or for writing new data.
Edit: I guess you would also need to change some spaces to hyphens. Shouldn't be too hard to write logic to accomplish that.
Something like this.
pseudo code:
$myinput = mysql_real_escape_string('101mxeGte-TS-200-10')
$query = " SELECT * FROM table1
WHERE REPLACE(REPLACE(f1, ' ', ''),'-','')
= REPLACE(REPLACE($myinput, ' ', ''),'-','') "
Alternatively you might write your own function to trim the data so it can be compared.
DELIMITER $$
CREATE FUNCTION myTrim(AStr varchar) RETURNS varchar
BEGIN
declare Result varchar;
SET Result = REPLACE(AStr, ' ','');
SET Result = ......
.....
RETURN Result;
END$$
DELIMITER ;
And then use this in your select
$query = " SELECT * FROM table1
WHERE MyTrim(f1) = MyTrim($myinput) "
have you ever heard of SQL's LIKE?
http://dev.mysql.com/doc/refman/4.1/en/string-comparison-functions.html
there's also regex
http://dev.mysql.com/doc/refman/4.1/en/regexp.html#operator_regexp
101mxeGte - TS 200-10
101mxeGte-TS-200-10
how about this?
SELECT 'justalnums' REGEXP '101mxeGte[[:blank:]]*(\-[[:blank:]]*)?TS[[:blank:]-]*200[[:blank:]-]*10'
digits can be represented by [0-9] and alphas as [a-z] or [A-Z] or [a-zA-Z]
append a + to make then multiple of that. perens allow you to group and even capture what is in the perens and reuse it later in a replace or something else.
RLIKE is the same as REGEXP.
I have a simple PHP script that loops over data in a CSV file, and adds the records to the database accordingly. One of my fields is a description field, but that description field itself has a comma (or multiple comma's) in it. It seems as though data for that field is only read up until the comma, however the next field is correct, so it is not as though the field after that is the remainder of the description, is is using the next column which is right.
Am I supposed to escape the comma? I am adding this data to a MySQL database, could that be where the issue is being caused?
My SQL query could be something like:
$description = $data[7]; //description column eg: "hello, my name is xxxxx, I am old"
INSERT INTO tblsomething (id, description) VALUES ($id, '$description');
The above statement only inserts the description as "hello" and nothing after the first comma it encounters.
Any ideas why this is?
Many thanks,
Simon
EDIT: This is solved, apologies to all as it was a silly mistake. It appears that the person who did the front end was creating arrays of content using the patter ',' to split the content. IT seems that the description - although supposed to be one array entry - was being split into multiple entries due to it containing comma's. This will be solved by using a more rare character like the pipe symbol to create our separators.
Thanks to all
Because it's not a CSV file. Fields in a CSV file that contain commas are supposed to be delimited by double quotes; this way the CSV functions in PHP will handle them properly.
I have a problem to insert detail from one table into another in MySQL table. The compatibility differs as the one table accepts commas to separate detail in a column and the other table accepts piping. What PHP code can I use to convert the commas into piping?
Ok guys let me re-phrase. I am doing a data dump from another db to my db, the problem is that the content formats of the two tables aren't compatible, meaning that my one table uses UPPERCASE and the other one uses camelCase, and that the one uses (,) without spaces behind or infront where the other table uses (|) with spaces on both sides. I'm lost and need help
Take a look at http://php.net/manual/en/function.strtr.php
I found the answer:
$replace = str_replace(", "," | ", $result2);