I need to run a process that will perform about 10,000 mysql inserts into a GoogleSQL instance. Normally, I would use a load data local infile query for this to avoid the script timing out, but my app is running in Google App Engine which has a read-only filesystem. Normally, when I need my GAE app to write to the filesystem, I can just use file names prefixed with gs:// and the php code will read/write to/from Google Storage transparently.
However, I doubt that MySQL will understand a file path of gs://path/to/my/file.
Is there another way that I can make a dynamically generated local file available in a Google App Engine environment so that I can load it into my GoogleSQL instance?
Otherwise, I feel like I'm going to need to build a looping ajax system to insert X rows at a time until it's gone through however many I need (10,000... 20,000, etc).
I know that I can put multiple values sets into a single insert to speed it all up and I'm planning to do that, but with the datasets as large as I'm dealing with, that still won't speed things up enough to avoid the timeouts consistently.
Related
I'm new at hosting with Google App Engine and haven't got my file caches to work yet any is therefore running a bunch of SQL queries that at least on my last dedicated server made more sense to have file cached.
My questions are:
Is it worth file caching on Google App Engine or is there some disadvantages of using a bucket to store TXT files for SQL queries that rarely change but are triggered a couple of times per second?
What's the most simple way of printing a file to a bucket in a flexible environment? I've tried just this code but it doesn't work. Am I entering the bucket path correctly and is there something I need to do before, like including some script, adding something to the app.yaml or something?
file_put_contents("gs://${my-test-project-xxxxxxxx.appspot.com}/cache/hello_caching.txt", "Text to be added");
I'm attempting to put together a mysql load balancer with a mockup php like script. Problem is i've been looking about countless variables in the database and cant find a variable of the current load on that server so I can pick the faster server to give the client.
Mysql is not aware of the server resource use, so what to do is to use for example Cacti, get the data from there and use that in your loadbalancing app.
Another way is just to use round robin and assume the systems will be pretty normally distributed over time.
Third option is to auto scale the number of slave servers using for example Kubernetes with NFS & ZFS for central storage and making snapshots of the database available on the slave nodes (for a read only solution
I am about to develop an application for the iOS devices, that will need to store information in a database on the web. For previous projects, i would just have PHP scripts, and my application would run those scripts and passing in the stuff to upload as _GET parameters, like this:
http://example.com?name=George&contents=iWillBePutIntoDB
However, this is not possible for my next project, as it will contain rather large amount of text, and i could exceed the maximum allowed length of a URL.
So, how do i go about doing this? I cannot access the MySQL database directly from my app, since my shared hosting provider doesn't allow it due to security reasons. So i can only access it using PHP scripts that are stored directly on the server.
In short: How do i upload large amount of text to a MySQL database that doesn't allow access from anywhere except files on the server itself (PHP scripts)?
Thanks everyone!
One obvious way would be using POST, instead of using GET.
But take it into your considerations that post is not without its limits.
I'm making a program which will be always gathering some data and putting them into a MySQL database, now i have two way to accomplish that and i wanted to knwo which one is better:
1- As i'm using Qt i can use QtSql module to connect directly to the DB and insert data
2- I can aslo write a PHP script using GET or POST varibles to insert data into the DB and just call the URL from my program with the suitable data.
I'm most worried about performance, as there will a lot of insertion all the time (about 100inserts / second). But the data size of each insertion is not that big, it will not exceed 10 charcaters.
I would point out that the Web server(where the php script will be stored), the DB server and the Server where the program will be running are all on the same local network
It depends on your environment. Is the MySQL DB running on your local PC? Then, QtSql should be perfectly fine.
If the DB is publicly accessible (eg. via the internet), then this would be a bad idea. You generally should avoid exposing MySQL DB servers directly to public network and instead provide access e.g. via locked-down interfaces such as your PHP one.
edit: From a pure performance-centric point of view, the QtSql solution should definitely be faster (though the only way to be sure is to benchmark it!). The PHP solution would have at least the following elements that generate overhead:
Data transfer over a network socket
Startup of the PHP interpreter (is it still the case that PHP is started once for each request? If so, this is going to be significant.)
I am writing an android application that needs to work both offline and online. In order to do that I need to get mysql database from external server and save that db to phones localdatabase so it can be read from there if user has no connection. Is it possible and if so, how can I do it?
Refer the following thread in which I have mentioned javascript frameworks which can be used to store data locally. The frameworks are using the standard local storage mechanism but they make it easier to store/retrieve and also make it portable across many platforms (this again depends on selection on the framework).
What to Use for PhoneGap Database Storage
Although you still need to load data from server.
This sounds like you need to write an API which does CRUD operations on a database.
There is no trivial way to simply do this directly with MySQL - you'll have to write one.
There are frameworks designed specifically for this, for example FRAPI: http://getfrapi.com/