I'm presently on a e-learning project using the custom php (not any framework or cms).For content of one page of my project i have to fetch about 1000 data from database,presently i'm using pagination on that page and displaying 100s of data every page.Now i'm thinking if i fetch all data from the database at a time and store it in xml and when user sweep between the pages of pagination the data will be fetched from the xml rather than database it may be good in the sense that it will reduce the database hits.But i have confusion that is the xml pursing may effect on my project execution time?If any better better idea please share with me.
My project's environment is like below
php 5
Mysql
Jquery
This still sounds inefficient since you have to still parse the xml.
I believe the most efficient way to do it (optimised for page views) would be to pre-generate the html of your lists.
That means everytime the database changes, you re-create the html, but only once.
Then all you do is simply serve that html from your web-server without any script executing.
Related
Currently working on a new web service that requires me to scrape a site every couple hours to save the data into my MySQL database.
My question is - how should my scraper run?
For now I see a few ways:
The cronjob runs a scraping script written in PHP, scrapes the data, and saves the data into a flatfile (i.e. csv), which then I setup a Controller to parse the data and have my Model save the data
The cronjob runs a scraping script written in PHP, scrapes the data, and immediately saves the data into my DB as each rows of data comes in
Of the two methods above, which one is better? If I am simply talking out of my ass, could you please suggest me a better way to:
Scrape the date
Save the data to my DB
Of the two options of saving scrapped data, if I were you, I would go after the second way. The reason is simply that it is easier to manage the scrapped data once they are already in DB -- it would save you the burden of generating and using temporary files.
Saving (appending new data) into a flat file may be faster than inserting into DB. But when time/performance is critical, you can either run your cronjob more frequently or run multiple copies of your cronjob (say, each of them scrapping different websites or different web pages).
I have a website that let's each user create a webpage (to advertise his product). Once the page is created it will never be modified again.
Now, my question: Is it better to keep the page content (only a few parts are editable) into a MySql database and generate it using queries everytime the page is accesed or to create a static webpage containing all the info and store it onto the server?
If I store every page on the disk, I may reach like 200.000 files.
If I store each page in MySQL database I would have to make a query each time the page is requested, and for like 200.000 entries and 5-6 queries/second I think the website will be slow...
So what's better?
MySQL will be able to handle the load if you create the tables properly (normalized and indexed). But if the content of the page doesn't change after creation, it's better if you cache the page statically. You can organize the files into buckets (folders) so that one folder doesn't have too many files in it.
Remember to cache only the content areas and not the templates. Unless each user has complete control over how his/her page shows up.
200.000 files writable by the Apache process is not a good idea.
I recommend using a database.
Database imports/exports are easier, not telling about the difference between the maintenance costs.
Databases are using caching, and if nothing is changed, they will pull up the last result, without running the query again. This doesn't stand, thanks JohnP.
If you want to redesign your webpage sometimes later you must be using MySQL to store the pages as you can't really change them (unless you dig into regexp) after making them static.
About the time issue - its not an issue if you set indexes right.
if the data is small to moderate then prefer static hardcoding ie. putting the data in the HTML, but if it is huge, computational or dynamic and changing you have no option but to use a connectivity to the Database
I believe that proper caching technique with certain attributes (long exp. time) would be better than static pages or retrieving everything from mysql everytime.
Static content is usually a good thing if you have a lot of traffic, but 5-6 queries a second is not hard for the database at all, so with your current load it doesn't matter.
You can spread the static files to different directories by file name and set up rewrite rules in your web server (mod_rewrite on Apache, basic location matching with regexp on Nginx and similar on other web servers). That way you won't even have to invoke the PHP interpreter.
A database and proper caching. 200.000 pages times, what? 5KB? That's 1 GB. Easy to keep in RAM. Besides 5/6 queries per second is easy on a database. Program first, then benchmark.
// insert quip about premature optimisation
Based on this tutorial I have built a page which functions correctly. I added a couple of dropdown boxes to the page, and based on this snippet, have been able to filter the results accordingly. So, in practice everything is working as it should. However, my question is regarding the efficiency of the proceedure. Right now, the process looks something like this:
1.) Users visits page
2.) Body onload() is called
3.) Javascript calls a PHP script, which queries the database (based on criteria passed along via the URL) and exports that query to an XML file.
4.) The XML file is then parsed via javascript on the users local machine.
For any one search there could be several thousand results (and thus, several thousand markers to place on the map). As you might have guessed, it takes a long time to place all of the markers. I have some ideas to speed it up, but wanted to touch base with experienced users to verify that my logic is sound. I'm open to any suggestions!
Idea #1: Is there a way (and would it speed things up?) to run the query once, generating an XML file via PHP which contained all possible results, store the XML file locally, then do the filtering via javascript?
Idea #2: Create a cron job on the server to export the XML file to a known location. Instead of using "Gdownloadurl(phpfile.php," I would use gdownloadurl(xmlfile.xml). Thus eliminating the need to run a new query every time the user changes the value of a drop down box
Idea #3: Instead of passing criteria back to the php file (via the URL) should I just be filtering the results via javascript before placing the marker on the map?
I have seen a lot of webpages that place tons and tons of markers on a google map and it doesn't take nearly as long as my application. What's the standard practice in a situation like this?
Thanks!
Edit: There may be a flaw in my logic: If I were to export all results to an XML file, how (other than javascript) could I then filter those results?
Your logic is sound, however, I probably wouldn't do the filtering in Javascript. If the user's computer is not very fast, then performance will be adversely affected. It is better to perform the filtering server side based on a cached resource (xml in your case).
The database is probably the biggest bottleneck in this operation, so caching the result would most likely speed your application up significantly. You might also consider you have setup your keys correctly to make your query as fast as possible.
I have to load some XML data (generated from a database, using PHP) into a flash slideshow.
The database data will change only when someone edit the website at it's backend.
In terms of loading speed and performance, which is best:
1) Generate the XML data dynamically from the database, each time the page is loaded;
2) Generate a .XML file whenever the database is updated, which will be read by the flash file.
The fastest is likely
3) use Memcached
Otherwise it is likely 2 because connecting to a database is usually a bottleneck and often slower than file I/O. But then again, you could simply benchmark it to see which works best for you. That's much better than assuming.
Also, have a look at this related question:
File access speed vs database access speed
#JapanPro He wouldn't need to write to the XML file when it was requested, just when someone saved something to the database. This would mean a much better load speed compared to pulling data from a database everytime.
Of course it depends how much data we're talking and whether it's worth writing to a file first. As #Gordon said, run some tests to see which works better for you
I think got for 1) Generate the XML data dynamically from the database, each time the page is loaded; is a good choice as its its normal as html. coz i think writing file always need more resource.
it depend on how your code is , if your code is processing alot of data every time, then writing file make sense
Hi I've got a tricky question (aren't they all tricky?)
I'm converting a database driven site that uses php, to a site being built with Dashcode.
The current site selects data held in a mySQL database and dynamically creates the page content. Originally this was done to reduce site maintenance because all the current content could be maintained and checked offline before uploading to the live database, therefore avoiding code changes.
In dashcode you can work from a JSON file as a datasource - which is fine, it works - except for the maintenance aspect. The client is not willing (and I understand why) to update several hundred lines of fairly structured JS object code when the database holds the data and is updated from elsewhere.
So - What's the best way to get Dashcode to link to the database data?
Where are you getting the the JSON from? Is that being generated from the original MySQL? Could you not generate the JSON from the MySQL and therefore use the original maintenance procedure prior to uploading to MySQL?
For my projects I usually create a php intermediate that when accessed logs into the MySQL database and phrases the results into xml in the body of the page. Just point daschcode to the php file in the data source. Parameters can even be passed into the php script through with GET through the url in the data source.