Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
All the answers about this question assume you're storing all of your user's data in one big file - and so they talk about how that is too slow.
Let's say I have thousands of users and store their data as JSON format in separate files (which I am currently doing), what is the downside to that - as opposed to setting up a proper database like Postgresql - which seems like overkill.
The speed is great on my current setup, but I am advised against doing this.
Since each user has their own separate file, there isn't really an issue of hundreds of people writing to the file at the same time (isolation).
Maybe it only matters for sites with millions of users?
In most systems, the users don't merely have to exist, they have to do stuff. And that stuff would generally be represented in a database. So you want the users to exist in the same system where the things they interact with exist.
What happens if your system crashes (power failure, for example) when a json file is half-way written out? Will you be left with a broken JSON file for that user? With databases, that should be taken care of automatically (you find either the old record, or the new one, not some truncation or mishmash). If you roll your own database, you will have to go some way out of your way to verify that you do this in a safe manner.
How do you name your user files? By the user's name? What if different people have the same name? What if their name has characters that can't be represented in file names? By an account number you assign? What happens if they forgot their account number and need to look it up by their human name? Do you then need to read and parse every user file to identify the correct one? Not that a database will magically make this free, but at least with a database you can just build an index with first having to invent and implement them.
You are basically reimplementing a database system from scratch, one feature at a time, as you discover the need for that feature. You can do it, sure. But why not use one that already exists?
Since each user has their own separate file, there isn't really an issue of hundreds of people writing to the file at the same time (isolation).
What if one person writes to one file at the same time from two different browsers (or tabs)?
There is no absolute right or wrong.
If you will never need to take care of concurrent access to the same record (file) or you don't need to search through your records or scale to multiple servers, the solution is fine and even faster than accessing a database.
I would just recommend to properly escape the user provided data, as JSON
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed last year.
Improve this question
Currently, I am working on a website and just started studying backend. I wonder why nobody uses JSON as a database. Also, I don't quite get the utility of php and SQL. Since I could easily get data from JSON file and use it, why do I need php and SQL?
ok! let assume you put the data in a JSON variable and store it in a file for all your projects.
obviously, u need to add a subsystem for getting back up, then you will write it.
you must increase the performance for handling a very large amount of data, just like indexing, hash algorithms, and... , assume u handle it.
if you need some API for working and connecting with a variety of programming languages, u need to write them.
what about functionalities? what if you need to add some triggers, store procedures, views, full-text search and etc? ok, you will pay your time and add them.
ok, good job, but your system will grow up and you need to scale it, can you do it? u will write abilities for clustering across servers, sharding, and ...
now you need to guarantee that your system will compatible with ACID rules, to keep atomicity, Consistency, Isolation, and Durability.
can you always handle all querying techniques (Map/Reduce) and respond with a fast and standard structure?
now it's time to offer very quick write speeds, it brings serious issues for you
ok, now proper your solutions for condition racing, isolation level, locking, relations and ...
after you do all this work plus thousands of many others, probably you will have a DBMS a little bit just like MongoDB or other relational and non-relational databases!
so it's better to use them, however, obviously, you can choose to don't to use them too, I admit that sometimes saving data in a single file has better performance, but only sometimes, in some cases, with some data, for some purpose! if you know what exactly you do, then ist OK to save data in a JSON file.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
Okay, basically, I have a PHP script written all up (it's an MMORPG, so we're clear).
What I'd love to be able to do is rather than writing new (massive) files that contain the exact same data, just in one big script (so as to be able to create more of an NPC aspect to the game), I'd rather just be able to send a request to open page.php using predefined get variables (i.e., collect=Y or attack=Y etc) that would be virtually identical to how a real player send the requests, and have the system open the file, run through the file, and make whatever queries to the database it needs to before closing it.
I'm confused on how fopen works to be honest, some things I've read make me believe the above is possible, others not so much.
Any help would be appreciated.
I'm going to go out on a limb, here, and try to solve your problem (as I understand it) rather than answer your question (as you have cast it).
Your fundamental problem is that you have treated PHP files as complete units of code, with input from the query string, processing specific to that file, and output back to the user. This violates the "single responsibility principle", because there are at least three top-level responsibilities here:
Processing user input and deciding on the appropriate action
Performing an action, including manipulation of database structures
Communicating the result of an action back to the user
These can all be broken down into smaller tasks - for instance, the nitty-gritty of connecting to the database should kept out of the more abstract actions, because changing how an enemy moves, and changing that enemy to be stored in a MongoDB document rather than a MySQL table should not require changes to the same code.
The solution, therefore, is to embrace structured programming, which in modern PHP (and many other languages) usually means embracing object-oriented programming. So, at a first level of organisation, you might have:
A class for looking at the query string, checking that it makes sense, and creating an abstract list of actions described by it.
Classes representing certain actions such as "Attack" and "Collect", which take details of what is being attacked or collected from the abstract list, and return a different abstract list detailing the results.
Classes representing the player, and enemies or objects within the game, which can be used by the action classes to calculate the outcome in different situations.
Classes for taking the result of actions and displaying them to the user.
Now, instead of saying "I need to create a query string, run the code the page would run, then take the output and use it somehow" you can say "I need to create an action list, run the appropriate actions, and use the result list somehow".
It may sound like that's a lot more work than just forcing PHP to run the existing code, but the power it gives you to create new combinations of existing functionality is not to be under-estimated.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I have a fixed (rarely changing) list of ~100 words. I want to display a random word on my HTML page on every reload. Should I hardcode the words as an array in the PHP script, or should I put them into a MySQL table and pull a random entry from there? What are the possible performance/maintainability considerations here?
It depends, if you ever want to easily manage these words or have someone else manage them, I would go for putting them into a database. Using a database has an extremely high overhead relative to a PHP array, although it is likely unnoticeable for a human if hosted locally.
I would not use anything other then a PHP array, database table, or text file though. I think that even a text file is a little bit extraneous and shouldn't be used - if you would want it in a text file it's probably just best to put it in a database.
My take is that it depends on these factors:
How rarely is "rarely"? Like once every year? Or maybe once every month?
How many requests are you getting, and how many are you predicting?
Do you have an established development/deployment cycle? Meaning, steps between changing some code and actually updating in production servers.
Do you have direct access to the databse? Or would you have to set up and admin tool in order to edit the list?
I would favor a non-MySQL scenario if you don't use the DB for anything else, or if you are getting millions of requests per day, so as not to add millions of queries for such a simple objective. Maybe using a local file with the words would suffice, if you have relatively straightforward access to the filesystem.
I wouldn't go for MySQL as its just not needed for a non-changing set of words. If you plan to change them whatsoever go for a CSV file [using implode() and explode() to manage it] or if you are very rarely or never changing them then a PHP Array would be best for performance, with 0 maintenance.
If you wanted to change the words via a nice interface you were going to write, I'd store them in MySQL. If they rarely change and it's just as easy to update your code as it is the database, then you might as well just store them in a PHP array.
I will say to use hardcode as php array instead of mysql and other connectivity part for these so it will be easy and you are saying you will use it rarely.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I finished creating an accounting web application for an organization using codeignter and mysql db, and I have just submitted it to them, they liked the work, but they asked me how they would transfer their old manual data to the new one online, so that their members would be able to see their account balances and contributions history.
This is a major problem for me because most of my tables make use of 'referential integrity' to ensure data synchronization and would not support the style of their manual accounting.
I know a lot of people here have faced cases like this and I would love to know the best way to collect users history, and I also know this might probably be flagged as not a real question, but I really really have to ask people with experience.
I would appreciate all answers. Thanks (And vote downs too)..
No matter what the case is, data conversions are very challenging and almost always time consuming. Depending on the consistency of the data in question, it could be a case that about 80% of the data will transfer over neatly if you create a conversion program using PHP. That conversion code in and of itself may be more time consuming than it is worth. If you are talking hundreds of thousands of records and beyond, it is probably a good idea to make that conversion program work. Anyone who might suggest there is a silver bullet is certainly not correct.
Here are a couple of suggested steps:
(Optional) Export your Excel spreadsheets to Access. Access can help you to standardize data and has tools in place to help you locate records which have failed in some way. You can also create filters in Access if you need to. The benefit of taking this step, if you are familiar with Access, is that you have already begun the conversion process to a database. As a matter of fact, if you so desire, you can import your MySQL database information into Access as well. The benefit of this is pretty obvious: You can create a query and merge your two separate tables together to form one table, which could save you a great deal of coding.
Export your Access table/query into a CSV file (note, if you find it is overkill or if you don't have Access, you can skip step 1 and simply save your .xls or .xlsx file to type .csv. This may require more legwork for your PHP conversion code but that is probably a matter of preference. Some people prefer to avoid Access as much as possible, and if you don't normally use it you will be wasting time trying to learn it just to save yourself a little bit of time).
Utilize PHP's built-in str_getcsv function. This will convert a CSV file into a PHP array.
Create your automated program to parse through each record. Based on the column and its requirements, you can either accept or reject records. You can then export your data, such as was done in this SO answer, back to CSV. You can save two different CSV files, one with accepted records, and one with rejected records.
With rejected records, which are all but inevitable when transferring from a spreadsheet, you will need to have a course of action. The simplest way for your clients is probably to give them a procedure to either manually import records into the database, if you've given them an interface to do so, or - probably simpler but requiring more back-and-forth - to update the records in Excel to be compliant with the new system.
Edit
Based on the thread under your question which sheds more light on what you are trying to do (i.e., have a parent for each transaction that is an accepted loan), you should be able to contrive a parent field, even if it is not complete, by creating a parent record for each set of transactions based around an account. You can do this via Access, PHP, or, more likely, a combination.
Conclusion
Long story short, data conversions take time. If you put the time in up front, it will be far easier to maintain a standardized series of information in the long run. If you find something which takes less time in the beginning, it will mean additional work for you in the long run in order to make this "simple" fix work over time.
Similarly, the closer you can get legacy data to conform to your new data, the easier it will be for your clients to perform queries etc. While this may mean that some manual entry will be required on the part of you or your client, it is better to inform the client of the pros and cons of each method fully and let them decide. My recommendation would always be to put extra work in at the front-end because it almost always ends up cheaper than having to deal with a quick fix in the long run, but that is not always practical given real world constraints.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I want to keep logs of some things that people do in my app, in some cases so that it can be undone if needed.
Is it best to store such logs in a file or a database? I'm completely at a loss as to what the pros and cons are except that it's another table to setup.
Is there a third (or fourth etc) option that I'm not aware of that I should look into and learn about?
There is at least one definite reason to go for storing in the database. You can use INSERT DELAYED in MySQL (or similar constructs in other databases), which returns immediately. You won't get any return data from the database with these kinds of queries, and they are not guaranteed to be applied.
By using INSERT DELAYED, you won't slow down your app to much because of the logging. The database is free to write the INSERTs to disk at any time, so it can bundle a bunch of inserts together.
You need to watch out for using MySQL's built in timestamp function (like CURRENT_TIMESTAMP or CUR_DATE()), because they will be called whenever the query is actually executed. So you should make sure that any time data is generated in your programming language, and not by the database. (This paragraph might be MySQL-specific)
You will almost certainly want to use a database for flexible, record based access and to take advantage of the database's ability to handle concurrent data access. If you need to track information that may need to be undone, having it in a structured format is a benefit, as is having the ability to update a row indicating when and by whom a given transaction has been undone.
You likely only want to write to a file if very high performance is an issue, or if you have very unstructured or large amounts of data per record that might be unweidly to store in a database. Note that Unless your application has a very large number of transactions database speed is unlikely to be an issue. Also note that if you are working with a file you'll need to handle concurrent access (read / write / locking) very carefully which is likely not something you want to have to deal with.
I'm a big fan of log4php. It gives you a standard interface for logging actions. It's based on log4j. The library loads a central config file, so you never need to change your code to change logging. It also offers several log targets, like files, syslog, databases, etc.
I'd use a database simply for maintainability - also multiple edits on a file may cause some getting missed out.
I will second both of the above suggestions and add that file locking on a flat file log may cause issues when there are a lot of users.