We have a heart monitor hooked up to to a TI msp430 microcontroller with a roving networks wifi module. I would like to send some type of a datastream to a webserver so that someone could monitor the data offsite. We were thinking that every half second we could send a datapoint to the php/mysql server about what the heart rate is. My problem is storing all that data. If I get one datapoint every second and create a new table entry for each datapoint, then I will start to get a lot entries in my table that contain very little data. I'm afraid this will slow things down significantly when we try to query the database and display the data causing our 'real time' data wouldn't be so 'real time'.
I was then thinking that every hour or something I could have the database batch up all the entries and turn it into one query. This seems to me like a bit of a hack, and I feel like there is a better way that I am missing.
Is there anyway I might be able to open up some type of a connection between the microcontroller to send the live data to the server and continuously write it to a file or something? Like a datastream of some type?
or
Can you keep session variables and whatnot when the microcontroller connects to the server? If we can, then it we could save all the data in a session variable until it gets to a certain size then write a chunk of data to the database with one entry and reset the session variable?
In no way will one data point per second slow down the performance of your database, even if you are running on a very limited server. This is what database abstractions are for, handling large amounts of data. It will actually be better than writing to a file in the long run since it is easy to select the last data point by id for use in your 'real time' application.
Related
So, I have situation and I need second opinion. I have database and it' s working great with all foreign keys, indexes and stuff, but, when I reach certain amount of visitors, around 700-800 co-current visitors, my server hits bottle neck and displays "Service temporarily unavailable." So, I had and idea, what if I pull data from JSON instead of database. I mean, I would still update database, but on each update I would regenerate JSON file and pull data from it to show on my homepage. That way I would not press my CPU to hard and I would be able to make some kind of cache on user-end.
What you are describing is caching.
Yes, it's a common optimization to avoid over-burdening your database with query load.
The idea is you store a copy of data you had fetched from the database, and you hold it in some form that is quick to access on the application end. You could store it in RAM, or in a JSON file. Some people operate a Memcached or Redis in-memory database as a shared resource, so your app can run many processes or threads that access the same copy of data in RAM.
It's typical that your app reads some given data many times for every single time it updates the data. The greater this ratio of reads to writes, the better the savings in terms of lightening the load on your database.
It can be tricky, however, to keep the data in cache in sync with the most recent changes in the database. In other words, how do all the cache copies know when they should re-fetch the data from the database?
There's an old joke about this:
There are only two hard things in Computer Science: cache invalidation and naming things.
— Phil Karlton
So after another few days of exploring and trying to get the right answer this is what I have done. I decided to create another table, instead of JSON, and put all data, that was suposed to go in JSON file, in the table.
WHY?
Number one reason is MySQL has ability to lock tables while they're being updated, JSON has not.
Number two is that I will downgrade from few dozens of queries to just one, simplest, query: SELECT * FROM table.
Number three is that I have better control over content this way.
Number four, while I was searching for answer I found out that some people had issues with JSON availability if a lot of co-current connections were making request for same JSON, I would never have a problem with availability.
I'm developing an Android app for salesman, so they can use their device to save their order. Specifically, every morning the salesman would go to the office and fetch the data for that day.
Currently, I can get the data by sending a request to php file, and like common practice we insert those data into sqlite in the Android so it can work offline. However, with current approach the device needs 6-8 seconds on getting the data and inserting those data to sqlite. As the data grow bigger I think it would make it slower. What I had found is that the process of inserting data into sqlite takes quite amount of time.
So, I've been thinking about dumping all data that is needed by the salesman into a sqlite file, so I could send only that file which I guess is more efficient. Can you please lead me on how to do that? Or is there any other way which is more efficient approach for this issue?
Note:
Server DB: Mysql
Server: PHP
You can do here different approach to achieve loading speed:
If your data can be pre-loaded with apk, you can just store those inside .apk and when user download app, it will be there, you just need to call remaining updated data.
If you need refreshed data every time, you can do call chunk of data from server in multiple calls, which will fetch and store data in database and update on UI.
If data is not too much, (I say, if there are 200-300 data, we should not consider it much more) you can do simple thing:
When you call network call for fetching data, you should pass that data objects to database for storing and at same time (before storing in db), just return the entire list object to Activity/Fragment, so it will have the data and you can see those data to user, in mean time, it will store in database.
Also, you can use no-sql in side sqlite, so you don't need to parse objects every time (which is costly compare to no-sql) and store all data in sql as entire object and whenever require, just fetch from db and parse it as per requirement.
Thanks to #Skynet for mentioning transaction, it does improve the process alot.. So I'll stay with this approach for now..
You can do something like so:
db.beginTransaction();
try {
saveCustomer();
db.setTransactionSuccessful();
} catch {
//Error in between database transaction
} finally {
db.endTransaction();
}
For more explanation: Android Database Transaction..
I have an android app, which needs to fetch new data from a server running a MySQL database. I add data to the database via a Panel which I access online on my domain.com/mypanel/.
What would be the best way to fetch the data on the client to reduce the overhead, but keep the programming effort as small as possible. It's not necessary for the client to get the latest database changes right after they have been updated, i.e. it would be okay if the client is updated some hours after the update.
Currently I thought of the following:
Add a column timestamp to the database-tables so that I know which changes have been made
Run some sort of background service on the client (in the app) which runs every X hours and then checks for the latest updates since the last successfull server-client synchronization
Send the time-gap to the server in which there haven't been any updates on the client anymore, using HTTP-POST
On the server, there will be some sort of MySQL SELECT-statement which considers the sent time-gap (if there is no time-gap sent from the client, just SELECT everything, e.g. in case of the first synchronization (full-sync)) --> JSON-Encode the Arrays -> Sent the JSON Response to the Client
On the client, take the data, loop row by row and insert into the local database file
My question would be:
Is there something you would rather do differently?
Or would you maybe send the database changes as a whole package/sql-file instead of the raw-data as array?
What would happen, when the internet connection aborts during the synchronization? I thought of the following to avoid any conflicts in this sort of process: Only after successfull retrieve of the complete server-response (i.e. the complete JSON-array), ONLY then insert the rows into the local database and update the local update timestamp to the actual time. If I've retrieved only some of the JSON rows and the internet connection gets interrupted inbetween (or app is being killed), I would NOT have inserted ANY of the retrieved rows into my local app-database, which means that the next time the background service is running, there will hopefully be no conflicts.
Thank you very much
You've mentioned database on client and my guess is that database is SQLite.
SQLite fully supports transaction, which means that you could wrap your inserts in BEGIN TRANSACTION and END TRANSACTION statements. A successful transaction would mean that all your inserts/updates/deletes are fine.
Choosing JSON has a lot of ups and a few downs - its easy for both client and server side. A downside I've been struggling in the past is with big JSONs (a few Mb). The client device have to download all the string and parse it at once, so it may run out of memory while converting the string to JSONObject. I've been there, so just keep that in mind as a possibility. That could be solved by splitting your update into pieces and marking each piece with its number and total number of pieces. Then the client device should know that it'd make a few requests to get all the pieces.
Another option you have is the good old CSV. You won't need the JSON includes, which will save your app some space. An upside is that you may parse and process the data line by line, so the memory impact would be very low. The obvious downside here is that you'll have to parse the data, which might be a problem, depending on your data.
I should also mention XML as an option. My personal opinion is that I'd use only if I really have to.
I always was sure it is better and faster to use flat files to store realtime visit/click counter data: open file in append mode, lock it, put data and then close. Then read this file by crontab once in a five minutes, store contents to DB and truncate file for new data.
But today my friend told me, that it is a wrong way. It will better to have a permanent MySql connection and write data right to DB on every click. First, DB can store results to memory table. Second, even we store to a table located on disk, then this file is permanently opened by it, so no need to find it on disk and open again and again on every query.
What do you think about it?
UPD: We talking about high-traffic sites, about million per day.
Your friend is right. Write to a file and then a cronjob sending to database every 5 minutes? That sounds very convoluted. I can't imagine a good reason for not writing directly to DB.
Also, when you write to a file in the way you described, the operations are serialized. A user will have to wait for the other one to release the lock before writing. That simply won't scale if you ever need it. The same will happen with a DB if you always write to the same row, but you can have multiple rows for the same value, write to a random one and sum them when you need the total.
It doesn't make much sense to use a memory table in this case. If your data doesn't need to be persisted, it's much simpler to use a memcache you probably already have somewhere and simply increment the value for the key.
If you use a database WITHOUT transactions, you will get the same underlying performance as using files with more reliability and less coding.
It could be true that writing to a database is heavy - e.g. the DB could be on a different server so you have network traffic, or it could be a transactional DB in which case every write has at least 2 writes (potentially more if indexes are involved), but if you're aware of all this stuff then you can use a DB, take advantage of decades of work by others and make your programming task easy.
I have a multiple devices (eleven to be specific) which sends information every second. This information in recieved in a apache server, parsed by a PHP script, stored in the database and finally displayed in a gui.
What I am doing right now is check if a row for teh current day exists, if it doesn't then create a new one, otherwise update it.
The reason I do it like that is because I need to poll the information from the database and display it in a c++ application to make it look sort of real-time; If I was to create a row every time a device would send information, processing and reading the data would take a significant ammount of time as well as system resources (Memory, CPU, etc..) making the displaying of data not quite real-time.
I wrote a report generation tool which takes the information for every day (from 00:00:00 to 23:59:59) and put it in an excel spreadsheet.
My questions are basically:
Is it posible to do the insertion/updating part directly in the database server or do I have to do the logic in the php script?
Is there a better (more efficient) way to store the information without a decrease in performance in the display device?
Regarding the report generation, if I want to sample intervals lets say starting from yesterday at 15:50:00 and ending today at 12:45:00 it cannot be done with my current data structure, so what do I need to consider in order to make a data structure which would allow me to create such queries.
The components I use:
- Apache 2.4.4
- PostgreSQL 9.2.3-2
- PHP 5.4.13
My recommendations - just store all the information, your devices are sending. With proper indexes and queries you can process and retrieve information from DB really fast.
For your questions:
Yes it is possible to build any logic you desire inside Postgres DB using SQL, PL/pgSQL, PL/PHP, PL/Java, PL/Py and many other languages built into Postgres.
As I said before - proper indexing can do magic.
If you cannot get desired query speed with full table - you can create a small table with 1 row for every device. And keep in this table last known values to show them in sort of real-time.
1) The technique is called upsert. In PG 9.1+ it can be done with wCTE (http://www.depesz.com/2011/03/16/waiting-for-9-1-writable-cte/)
2) If you really want it to be real-time you should be sending the data directly to the aplication, storing it in memory or plaintext file also will be faster if you only care about the last few values. But PG does have Listen/notify channels so probabably your lag will be just 100-200 mili and that shouldn't be much taken you're only displaying it.
I think you are overestimating the memory system requirements given the process you have described. Adding a row of data every second (or 11 per second) is not a hog of resources. In fact it is likely more time consuming to UPDATE vs ADD a new row. Also, if you add a TIMESTAMP to your table, sort operations are lightning fast. Just add some garbage collection handling as a CRON job (deletion of old data) once a day or so and you are golden.
However to answer your questions:
Is it posible to do the insertion/updating part directly in the database server or do I >have to do the logic in the php script?
Writing logic from with the Database engine is usually not very straight forward. To keep it simple stick with the logic in the php script. UPDATE (or) INSERT INTO table SET var1='assignment1', var2='assignment2' (WHERE id = 'checkedID')
Is there a better (more efficient) way to store the information without a decrease in >performance in the display device?
It's hard to answer because you haven't described the display device connectivity. There are more efficient ways to do the process however none that have locking mechanisms required for such frequent updating.
Regarding the report generation, if I want to sample intervals lets say starting from >yesterday at 15:50:00 and ending today at 12:45:00 it cannot be done with my current data >structure, so what do I need to consider in order to make a data structure which would >allow me to create such queries.
You could use the a TIMESTAMP variable type. This would include DATE and TIME of the UPDATE operation. Then it's just a simple WHERE clause using DATE functions within the database query.