I'm learning SQLite (with PHP), coming from a MySQL background.
I have a big project where I need to store huge csv files in a database, to enable quick searching, indexing, etc. The application always needs to use only one csv at a time, and there can potentially be thousands of csv-files, so I thought putting all data into one single database (MySQL) is not such a good idea. That's why I'm looking for a SQLite solution now.
Now my question is, how would you replicate the database structure with SQLite each time?
Do I need an empty 'dummy' database file, that I copy each time a new csv-file is uploaded?
Or would a script that generates the database be better?
And also, how would I deal with an eventual change in the database schema (in a newer version of the software) ? All existing database files would have to be changed then.
Related
I just took over a pretty terrible database design job, which heavily use comma separated value to store data. I know I know, it is hell.
The db is mysql, currently accessing it using MySql Workbench.
I already had idea in mind what to remove, and what new relations table needed.
So, my question is, how shall I proceed by migrating comma separated data to the new table? Any tools specialize for normalizing database?
Edit:
The server code is in PHP.
Define you new tables and attributes first.
Then, use PHP or Python or your favorite language with MySQL calls and write a 1 time converter which loops and reads the old table(s) and records and inserts the proper records into the new tables.
It appears you are looking for standard practices. There are varying degree of denormalized databases out there. The ones I have come across have been normalized with custom code and tools.
SQL Server Integration Services (SSIS) can be used for some case. In your case, I'd build a script for the migration that involves:
creation of normalized tables
creating stored procedure or PHP script(s) to read data from denormalized table, transform it and load it into normalized table
creating a log table or log file
performing the migration in sandbox; write logs while doing so
version control the script
correct the proc/script as needed
create another sandbox
run the full script on sandbox
if successful, run the full script on prod (with logging)
SSIS is used for ETL in many organizations; it's standard tool for Microsoft BI stack and can also be used to migrate data between non-Microsoft DBs also.
Open Source ETL tool called Talend might also help in transforming your data. I personally believe that a PHP script will be the fastest and easiest to manipulate data.
I need to load XML data from an external server/url into my MySQL database, using PHP.
I don't need to save the XML file itself anywhere, unless this is easier/faster.
The problem is, I will need this to run every hour or so as the data will be constantly updated, therefore I need to replace the data in my database too. The XML file is usually around 350mb.
The data in the MySQL table needs to be searchable - I will know the structure of the XML so can create the table to suit first.
I guess there are a few parts to this question:
What's the best way to automate this whole process to run every hour?
Whats the best(fastest?) way of downloading/ parsing the xml (~350mb) from the url? in a way that I can -
load it into a mysql table of my own, maintaining columns/ structure
1) A PHP script can keep running in background all the time, but this is not the best scenario or you can set a php -q /dir/to/php.php using cronos (if running on linux) or other techniques to makes server help you. (You still need access to server)
2) You can use several systems, the more linear one, less RAM consuming, is if you decide to work with files or with a modified mySQL access is opening your TCP connection, streaming smaller packages (16KB will be ok) and streaming them out on disk or another connection.
3) Moving so huge data is not difficult, but storing them in mySQL is not waste. Performing search in it is even worst. Updating it is trying to kill mySQL system.
Suggestions:
From what i can see, you are trying to synchronize or back-up data from another server. If there is just one file then make a local .xml using PHP and you are done. If there are more than one i will still suggest to make local files as most probably you are working with unstructured data: they are not for mySQL. If you work with hundreds of files and you need to search them fast perform statistics and much much more... consider to change approach and read about hadoop.
MySQL BLOOB or TEXT columns still not support more than 65KB, maybe you know another technique, but i never heard about it and I will never suggest to do so. If you are trying it just to use SQL SEARCH commands you took the wrong path.
I'm currently building a web-app which displays data from .csv files for the user, where they are edited and the results stored in a mySQL database.
For the next phase of the app I'm looking at implementing the functionality to write the results into ** existing .DBF** files using PHP as well as the mySQL database.
Any help would be greatly appreciated. Thanks!
Actually there's a third route which I should have thought of before, and is probably better for what you want. PHP, of course, allows two or more database connections to be open at the same time. And I've just checked, PHP has an extension for dBase. You did not say what database you are actually writing to (several besides the original dBase use .dbf files), so if you have any more questions after this, state what your target database actually is. But this extension would probably work for all of them, I imagine, or check the list of database extensions for PHP at http://php.net/manual/en/refs.database.php. You would have to try it and see.
Then to give an idea on how to open two connections at once, here's a code snippet (it actually has oracle as the second db, but it shows the basic principles):
http://phplens.com/adodb/tutorial.connecting.to.multiple.databases.html
There's a fair bit of guidance and even tutorials on the web about multiple database connections from PHP, so take a look at them as well.
This is a standard kind of situation in data migration projects - how to get data from one database to another. The answer is you have to find out what format the target files (in this case the format of .dbf files) need to be in, then you simply collect the data from your MySQL file, rearrange it into the required format, and write a new file using PHP's file writing functions.
I am not saying it's easy to do; I don't know the format of .dbf files (it was a format used by dBase, but has been used elsewhere as well). You not only have to know the format of the .dbf records, but there will almost certainly be header info if you are creating new files (but you say the files are pre-existing so that shouldn't be a problem for you). But the records may also have a small amount of header data as well, which you would need to write to work out and each one in the form required.
So you need to find out the exact format of .dbf files - no doubt Googling will find you info on that. But I understand even .dbf can have various differences - in which case you would need to look at the structure of your existing files to resolve those if needed).
The alternative solution, if you don't need instant copying to the target database, is that it may have an option to import data in from CSV files, which is much easier - and you have CSV files already. But presumably the order of data fields in those files is different to the order of fields in the target database (unless they came from the target database, but then you wouldn't presumably, be trying to write it back unless they are archived records). The point I'm making, though, is you can write the data into CSV files from the PHP program, in the field order required by your target database, then read them into the target database as a seaparate step. A two stage proces in other words. This is particularly suitable for migrations where you are doing a one off transfer to the new database.
All in all you have a challenging but interesting project!
I have a huge Database on the server and i need that DB in SQLite so that i can use it in my application in android as well as in ios app.
I got one solution for this when i go to phpmyadmin and select my db on server i exported the Tables one by one into CSV file and then imported those in my SQLite Browser one by one to get all the tables (And Then corrected column names and type manually by editing every table columns).
This way i made it as a .sqlite DB to be used in the app.
But i want to know more on these points below :
Is their some kind of a backend application that most developers use to convert their DB into SQLite DB. (If yes then what kind of stuff do they use)
Is their any PHP script that can do this stuff. (If yes then what script is used and how ?).
Is their any other simple way to deal with this problem of getting SQLite DB from the server. (If yes then what are the possible ways to do this ?).
Can any one get me some idea about this ?
I am planning to start a site in which the content is generated by the users. What is the best method to save the data submitted by user?
Is the method of saving the data in phpmyadmin database a good idea considering that the data submitted by users is large similar to a blog post. Btw I'm good in working with php and mysql but I'm not sure whether it is a good method.
By 'phpmyadmin' database I assume you just mean a MySQL database.
Since your user data is basically a 'blog post' - basic text and HTML, and you'll most likely be storing username, posting dates, titles, and the like as well -- a MySQL database is a fine place to store it.
If you're using standard shared hosting, your options are pretty much a relational database (MySQL) or a flat file. Between those two choices, a relational database is the better option.
Not sure just what you're asking here - If you're thinking of actually trying to save a .jpg file and text as blobs physically stored in the database, don't. Not if you intend to have a lot of users all uploading stuff. Look into saving it on your server in a folder for the user or better yet to the cloud - it'll be cheaper in the long run and save you tons of anguish with a corrupt database.