We need to import a MySQL dump from another website to ours. We've been trying SimpleXML, XML DOM etc, but the file is so huge it's crashing our server. We looked into BigDump, but that doesn't handle the XML import. Every tag in our XML file is called <field name="something"> In <table_something> tags, which I haven't seen before - usually it's descriptive custom tags. This is probably because I haven't done much database importing before now,
What we would like is some way to make our PHP import this huge file. It needs to be freshly updated every night so I'm thinking of dropping the tables and importing fresh unless there's a way to search for differences but I wouldn't know how.
Can anyone help with this? What would be the standard procedure for achieving these results?
Moving a database is usually done outside of PHP, via command line script:
Dump db to a file
tar the file
FTP to new server
Then on the new server
untar the file
Import to mysql
Do you have shell access?
If you have to do it via PHP, you'll need to split up the dump into lots of tiny files and import them one at a time. Depending on the size of your database, this could take a really long time.
When doing the dump, minimize the file size by not using XML, stripping comments, etc.
Related
At the moment I am doing a mass interface of files/data and some files are in XLS format, which I need to normalize them into csv (so basically, convert XLS to CSV files)
The problem is that PHPExcel (and similar libraries) load the entire sheet data at once thus exhausting memory.
So far I tried various libraries (in the meantime negotiating to have the data in csv though no luck so far)
I am running my tests on various large file sizes, my memory allocation is set properly before and after my script runs using ini_set etc.
Is there a way that I can read an xls line by line or in chunks (like fgetcsv or fread) please?
I am programming this so it can work with any filesize (even if it takes ages to run) as this is a fully automated system.
PS: I checked this post and various others already
Reading an Excel file in PHP
Possible ways...
Get help from other languages. e.g. find a Python excel library and use it. Then call Python from PHP.
Modify the source code of those Excel readers
Use a command line tool to convert excel to csv, e.g. Pandoc maybe, and use the csv in PHP
Since xls file is nothing but a zip file, maybe it can be unzipped and found the values
First decompose one xls into many small xls files via non-PHP solution, e.g. VBA in excel, then read each of them.
I want to read an Excel file with PHP row by row because reading the entire file at once cause memory overflow.
I have searched a lot, but no luck until now.
I think PHPExcel library can read chunks of an excel file, when you implement the filter class, but each time it gets this chunk it reads the entire file, which is impossible in huge .xls files because of the time it will take.
Any help ?
This may be something that is totally out of question, but from the information that I get from your question the following seems like an obvious option, at least something to consider ...
I get the impression that this is a really big file that needs to be accessed often. So, I would just try to import its data in a database.
I guess there is no need to explain that databases are masters in performance and caching.
And it is still possible to export the contents of the database to an excel file afterwards.
MySql works great with PHP and is certainly easier to access than an excel file. Most php hosting providers offer a MySql database by default with a PhpMyAdmin management tool.
How to do it:
If you have PhpMyAdmin installed, then you can follow these simple steps.
If you have command-line access to the server then you can even import the file from commandline directly to a MySql database.
If the only thing you need from your read Excel file is data, here is my way to read huge Excel files :
I install gnumeric on my server, ie with debian/ubuntu :
apt-get install gnumeric
Then the php calls to read my excel file and store it into a two dimensionnal data array are incredibly simple (dimensions are rows and cols) :
system("ssconvert \"$excel_file_name\" \"temp.csv\"");
$array = array_map("str_getcsv", file("temp.csv"));
Then I can do what I want with my array. This takes less than 10 seconds for a 10MB large xls file, the same time I would need to load the file on my favorite spreadsheet software !
For very huge files, you should use fopen() and file_getcsv() functions and do what you have to do without storing data in a huge array to avoid storing the whole csv file in memory with the file() function. This will be slower, but will not eat all your server's memory !
I am having issues with including really large files and so have used ini_set('memory_limit', '-1');
but still cannot include a file that is just over 1GB. What should I do?
You will have to reconsider your architecture; PHP's include function was not designed to handle such large files; it was designed to include and evaluate a PHP code file. Without knowing what data the file actually holds, it's hard to say more; but it seems very unlikely that this file actually only holds PHP code; it sounds like you're trying to read a lot of information which is encoded in a PHP-like format.
You should e.g. try to read the file in chunks or lines instead of using include.
The only possible reason you would be trying to load such a file is to probably import it into a database. Each database has a file loader that is meant for this purpose. Mysql has LOAD INFILE specifically to deal with this issue. I'd recommend getting the data in your database first, then use PHP or MySQL to transform the data to your needs. Using PHP to parse through a 1GB file is probably not the best usage of PHP or your resources.
Is it possible to write .MDF files using PHP. I have a simple HTML table - x columns, y rows that I wish to convert into a MS Access dataset. There are no foreign keys or anything exotic, just a number of rows and columns.
I have Google searched but the only advice I seem to be able to find is to save it as a .CSV. Unfortunately this isn't acceptable for my client. They want to simply be able to double click the icon and have the database load up in Access (Seems you have to import csv files and can't just double click them).
Those are actually two different problems:
1) Parsing HTML to separate values
Since you seem to have found a way to export to csv, I won't be answering this. You already got your parser then.
2) Create an mdb file.
You can use odbc to talk to Access, that is, if you are in Windows. That means that PHP must be installed on a Windows machine. I'm not aware of any Linux drivers for Access..
You can use odbc_connect to connect to the database:
$conn = odbc_connect("dbname","" ,"");
From there, working with the database is pretty similar to MySQL as long as you use the ODBC functions.
The only problem is creating the actual MDB. This cannot be done in ODBC, you'll need the JET engine. It may be possible to create an MDB file from PHP, but it may be easier to create an empty MDB once, and copy that file from PHP whenever you need a new database.
Jackcess is a pure Java library for reading from and writing to MS Access databases. It is part of the OpenHMS project from Health Market Science, Inc. . It is not an application. There is no GUI. It's a library, intended for other developers to use to build Java applications.
http://jackcess.sourceforge.net/
But an MDF file is not a standard Access file extension. MDB and MDE are while MDW is for user level security.
Have not tested this but this might help you:
http://devzone.zend.com/article/4065
Note that active development of MDB Tools has moved from Sourceforge to Github
https://github.com/brianb/mdbtools
I am trying to create an outlook distribution list with php but I am having a hard time finding any resources or instructions on how to do so. Is it even possible to do this?
If my memory serves me correct, you can create an XML based contact list that you then import into Outlook. My best suggestion would be to export some contacts from outlook, and look at the XML and then recreate that XML structure via PHP and your contacts.
EDIT - it seams that you can create an CSV file and import that as a distribution list.