I'm doing my student work. Please help me understand in what direction should I move.
So, I have a big MySQL database (31.3Gb, 130 millions rows) on my PC. I have written a simple html page (interface for my program). I'm using PHP for connecting to my database, AJAX for retrieve data from a server asynchronously and OpenServer (local server for testing).
How it works: the user selects some filters on my html page, clicks the search button, and it displays the main table.
Everything works well, but what should I use to connect my project to Hadoop? It is a mandatory requirement.
Maybe HortonWorks/Cloudera/Docker?
It will be very useful if you send links on some useful guides.
The main criterion is simple to configure and install. Also, I have only 4Gb of RAM on my laptop, so most likely I have to use old versions of tools. HortonWorks 2.1 was successfully installed even on my virtual machine (Ubuntu 14.04)
In advance thank you everyone for your help!
You can use Sqoop import to push data from MySQL to Hive.
Then, you can query via HiveServer2.
For getting Hadoop and other components, it is best to use HDP or CDH sandbox. If your machine resources do not permit using the sandbox, consider using AWS.
Once you have data in Hive, you can point your application to query Hive and serve results, instead of going to MySQL.
Related
Does it exist any difference between MySQL and PHPMyAdmin? arent they DBMS? I really dont find the difference between these two, I read the concepts so I know they are not the same but I cant spot the differences or when should use one instead of another one. Can someone explain it to me?
MySQL is a database server. It stores data, and runs queries on it that clients send to it.
PHPMyAdmin is a database client. It provides a user interface (written as a web app) that sends queries to the database server and displays what it sends back.
The difference is rather akin to the difference between Google Chrome (although Chrome is a desktop app rather than a web app) and Apache HTTPD.
My plan is to store data in my local MySQL database and after a time interval it will update my live MySQL database remotely. Is it possible?
I'm planning a inventory management script in PHP MySQL. I will install web application locally to my clients and it will backup local data to live server via API or any library.
Can any one suggest me any library for this all.
Thanks is advance.
You can achieve this with different methods:
Database Replication (recommended solution), it almost handles everything perfectly. You can read different tutorials available on internet to use it.
Scheduled PHP script to sync data on your specified occasions. There are several packages available for this, i.e. https://github.com/mrjgreen/db-sync. For scheduling you can use CronTab or supervisor or etc.
But I would personally recommend you to use replication, since it is a native DBMS solution for such scenarios.
I have a simple MySQL database (one table with 12 rows of data and 5 columns) sitting on the web-server of my host provider.
On my home PC I create the data programmatically and store it in a free version of SQL Server (on my home PC). I would like to "upload" this data to the MySQL db in real time (or as close as I can get) over the internet (I'm assuming this is the only way to connect the pipes).
I know that opening up a MySQL database to a remote internet connection probably is not a "secure" thing to do, but the resulting data table will be publicly available anyway via an "app" so I'm not too worried about that (I suppose a hacker could "overwrite" my data with their own if they were both industrious and inclined) but I think the risk/reward is so small its not a major concern.
Anyway, what is the easiest way to do this with some semblance of security? I only know how to program in VB (I did a little HTML and ASP back in the day, but that was a long time ago). I could learn a few lines of code in another language if need be.
I do not have a static IP, and I've never actually interacted with a MySQL database before (only SQL server, so my MySQL knowledge/ familiarity is zero...but a db is a db, so how hard can it be?). Because of my home network firewall, I can't allow connections "in". I will have to make the connection to the MySQL db "out" from my home PC --> to the hosted database.
Ok this problem is not actually super simple.
What you will find is most shared hosting providers do not allow just any IP to access their databases.
Solution? set the IP for your computer of course! BUT.....you are probably on home internet connection so your IP address can CHANGE (if you have a static IP you are a lucky person!)
So the best way - create a mini-API!
Basically, you want to post your data to a script (with some security of course) that then inserts this data into the database.
It is a lot of work but having done all this before it seems to be the only way unless you have a dedicated server / advanced access privileges!
You could take a look at WAMP for your home pc. It's simple to use.
And then you should take a look at Mysql remote connections(some details here)
I would try this:
At your local computer install MySQL server, there's a free community
edition available for download, try the web installer since its more lightweight. Use the
custom installation and make sure MySql Workbench is selected too.
Workbench has a migration tool for the most common databases, Try this locally, so you can tell if all your data is correctly migrated from your local SQL Server to a MySQL db and there are no data losses in the process.
Then, You could probably connect through Workbench to your online MySQL db
and migrate your data to it directly from your just created local db. In case you cannot connect, make a backup of your local db and send the files to
your server by ftp or similar process. Then, simply restore DB from the backup file on your
online server.
Hope this helps!
i had 2 databases, one run in my localhost which is offline and another is in server which is online, and what i want is after i insert some data in my localhost offline, i can click a button in my offline which is sync the database and update it on the server database which is online, and anyone of you know how to do it? Im new in php and mysql, hope you all can help, thanks!!
Doing synchronization like this is, I'm sorry to say, not a beginner activity.
If you were more experienced I would recommend things like a transaction table - where you record each action, then replay it on the server (aka slave).
MySQL can also do that automatically see: http://dev.mysql.com/doc/refman/5.0/en/replication.html
But none of it is simple to setup.
If you don't mind deleting and recreating the database fresh each time, then you maybe be able to use mysqldump to create a full export of your database, then load it on the server.
What would be a better choice for making a database driven
Adobe AIR(Desktop) application?
Using PHP+MySql with AIR
OR
Using SQLite
If I choose SQLite, then I cannot reuse my code for
an online application.
If I choose 1, I have to block few port numbers on User's machine.
Also, I am using XAMPP for providing the user with PHP and MySql, so
XAMPP opens up a command window as long as it's running. And, users
get confused about what's that window for? It makes the end user
experience slightly confusing.
I'd definitely use SQLite as its included in Air.
May I suggest; write your code in two sections. The UI which uses a JSON feed to populate itself, and the API to provide the JSON data. When it comes to port the application to the web you can use the same UI but with a rewritten API.
Whatever you do, don't open up a command window while the program is running. If you do that, your customers will uninstall like there's no tomorrow.
As far as mysql vs sqlite, the standard approach is - if it communicates remotely, feel free to use mysql, but if you're installing the db on the client, you should use an embedded standalone db (sqlite).
How complex do you expect your app to be that you can't use sqlite (besides not being able to reuse some of the code that you mentioned)?
If XAMPP is too confusing for your client, install Apache and MySQL as standalones. It's essentially the same thing and you'll have more control over what's running in Apache/MySQL. Plus you won't get an annoying command window (though, to be quite honest, I don't recall a window that I couldn't minimize to the tray when I ran XAMMP).
My suggestion is use Sqllite as your local database and writes a synchronization API that will synchronize the local sqllite database with the server side database-MySql. So according to your client you can use the system. If the client is standalone then Sqllite will serve otherwise the MySql will serve. Only thing you have to decide in both this is how to use the synchronization api.
Just check the Sample Application