I have a website live cricket scores , in which dynamically i am controlling the news section.
I have my own custom build CMS system with PHP, where admin will add the news to the web portal.
If i generate the Sitemap, all dynamically created pages wont be added to the sitemap,
is this a good practice or do we need to add the dynamically created links in sitemap?
if yes, can you please share how we can add dynamic links?
One more observation, I have made, whatever the news which is added getting cached within 4 Hrs in google.
Please share your thoughts, thanks in advance
If the pages are important, then you should add them to the site map so they can be indexed for future reference. However, if the pages are going to disappear after the match, then I wouldn't put them on the site map as they may get indexed then disappear, which may have a negative impact on your search engine rankings.
You can add these dynamic pages to a site map in a couple of ways:
Whenever a new dynamic page is created, re-create your site map. Do this by looking through the database for the pages which will be valid and writing them out into an XML site map file.
When a new page is created, read the current XML site map, and insert a new entry into the relevant place.
I would say the easiest option is option 1 as you can quickly and easily build a site map without having to read what you already have. That option also means that when you remove a one of the dynamic pages, it will be removed from the site map when it is re-built without the need to read through what you have, find the entry and remove it.
Google code has a number of different options for you, some of which you can download and run, others look like they need implementing within your own code.
Yes, if these pages content needs to be referenced by search engines, of course they have to be in sitemap.
I worked on a lot of ebusiness website and of course, almost 99% of pages where dynamically generated, almost 1000 product pages versus the 3 sales conditions & legal static pages.
So the sitemap itself was dynamic and regenerated every 15 minutes (to avoid dumping the whole product base each and running thousands of queries each tim the sitemap is called).
You can use a sort of separate script to do this : I would do one static part template if you have static page, and one other embedding the dynamically generated urls.
It would be easier if you CMS already embed url management (or routing) system.
Related
I am creating a site and want to have individual pages for each row in a database table. The information on each page is fairly useful and comprehensive, and it would be really nice if Google could index them.
My initial thought was to just create a single PHP template page and pull the correct information for whatever the user is looking at, but my fear is that search engines won't be able to index all of the pages.
My second thought was to batch-create/automate the process of creating the individual pages as html files (for the 2000+ rows in the table), because then I would be guaranteed that they'd be crawled. However, if I ever needed to make a change to the design, I'd have to re-process them all. Kind of a pain...
My final consideration was to just pick a page in my site and list all of the possible php pages in a hidden div, but I wasn't sure if search engines can index from that. I assume they just pull from the HTML, so it'd be able to find it, right?
Any suggestions? I would love it if I can just create a single page that populates based on what they user clicks, but I want them to be indexed.
Search engines can index dynamic pages so using one PHP file to create thousands of unique product pages will be fine for SEO. After all, each page/product will have a unique URL and will be seen as a unique page as a result. All you need to do is link to your product pages within your website and/or submit an XML sitemap so you can be sure they are found and indexed.
By linking your pages, I literally mean link to your product pages. Search engines find new content primarily through following links. So if you want your product pages to be found you need to link to them. Using form based search is not a good way to do it as search engines generally don't play to well with forms. But there are lots of way to make links to your pages including HTML sitemaps and product category pages which then can link to products in that category. Really, any way yo u an get a link to your product pages is a good way to help ensure they are found by the search engines.
You don't have to post links on invisible DIV!
Just create the page and have parameterized content fetching.
You can include the pages in the XML sitemap and submit to Google or you can include your page urls in the HTML sitemap too.
I would like to know from others' experience the best way to create sitemaps with Codeigniter. I have looked at some plugins/libraries, but all check the database for the pages. What happens if some pages on the site are static and not dynamic?
Is there any way to crawl the site using PHP and creating an XML file with the results?
A tool I have used previously for my projects is http://enarion.net/tools/phpsitemapng/download/
Which is a free tool for creating sitemap and allows functionality such as cron jobs.
What is my next step? How can I achieve this?
Well, you're problem lies in the fact that you have both dynamic and static pages. So, a crawler would work, but you'd have to generate a list of links to all dynamic pages. Then, you're crawler could hit that list and have access to all dynamic pages, and then hit directories where you have your static pages.
However, the docs on the phpsitemapng that you mention state that they will crawl a live website. So, if you have links to all of your pages accessible from those pages, then that will do what you need.
Scans files on website (slower, but will also find dynamic generated files and links)
I am creating a classifieds website.
Im storing all ads in mysql database, in different tables.
Is it possible to find these ads somehow, from googles search engine?
Is it possible to create meta information about each ad so that google finds them?
How does major companies do this?
I have thought about auto-generating a html-page for each ad inserted, but 500thousand auto-generated html pages doesn't really sound that good of a solution!
Any thoughts and idéas?
UPDATE:
Here is my basic website so far:
(ALL PHP BASED)
I have a search engine which searches database for records.
After finding and displaying search results, you can click on a result ('ad') and then PHP fetches info from the database and displays it, simple!
In the 'put ad' section of my site, you can put your own ad into a mysql database.
I need to know how I should make google find ads in my website also, as I dont think google-crawler can search my database just because users can.
Please explain your answers more thoroughly so that I understand fully how this works!
Thank you
Google doesn't find database records. Google finds web pages. If you want your classifieds to be found then they'll need to be on a Web page of some kind. You can help this process by giving Google a site map/index of all your classifieds.
I suggest you take a look at Google Basics and Creating and submitting SitemapsPrint
. Basically the idea is to spoon feed Google every URL you want Google to find. So if your reference your classifieds this way:
http://www.mysite.com/classified?id=1234
then you create a list of every URL required to find every classified and yes this might be hundreds of thousands or even millions.
The above assumes a single classified per page. You can of course put 5, 10, 50 or 100 on a single page and then create a smaller set of URLs for Google to crawl.
Whatever you do however remember this: your sitemap should reflect how your site is used. Every URL Google finds (or you give it) will appear in the index. So don't give Google a URL that a user couldn't reach by using the site normally or that you don't want a user to use.
So while 50 classifieds per page might mean less requests from Google, if that's not how you want users to use your site (or a view you want to provide) then you'll have to do it some other way.
Just remember: Google indexes Web pages not data.
How would you normally access these classifieds? You're not just keeping them locked up in the database, are you?
Google sees your website like any other visitor would see your website. If you have a normal database-driven site, there's some unique URL for each classified where it it displayed. If there's a link to it somewhere, Google will find it.
If you want Google to index your site, you need to put all your pages on the web and link between them.
You do not have to auto-generate a static HTML page for everything, all pages can be dynamically created (JSP, ASP, PHP, what have you), but they need to be accessible for a web crawler.
Google can find you no matter where you try to hide. Even if you can somehow fit yourself into a mysql table. Because they're Google. :-D
Seriously, though, they use a bot to periodically spider your site so you mostly just need to make the data in your database available as web pages on your site, and make your site bot-friendly (use an appropriate robots.txt file, provide a search engine-friendly site map, etc.) You need to make sure they can find your site, so make sure it's linked to by other sites -- preferably sites with lots of traffic.
If your site only displays specific results in response to search terms you'll have a harder time. You may want to make full lists of the records available for people without search terms (paged appropriately if you have lots of data).
First Create a PHP file that pulls the index plus human readable reference for all records.
That is your main page broken out into categories (like in the case of Craigslist.com - by Country and State).
Then each category link feeds back to the php script the selected value regardless of level(s) finally reaching the ad itself.
So, If a category is selected which contains more categories (like states contain cities) Then display the next list of categories. Else display the list of ads for that city.
This will give Google.com a way to index a site (aka mysql db) dynamically with out creating static content for the millions (billions or trillions) of records involved.
This is Just an idea of how to get Google.com to index a database.
How can I make it so that content from a database is available to search engines, like google, for indexing?
Example:
Table in mysql has a field named 'Headline' which equals 'BMW M3 2005'.
My site name is 'MySite'
User enters 'BMW M3 2005 MySite' in google and the record will show up with results?
Google indexes web pages, so you will need to have a page for each of your records, this doesn't mean to say you need to create 1,000 HTML pages, but following my advice above will dynamically / easily provide a seemingly unique page for each product.
For example:
www.mydomain.com/buy/123/nice-bmw-m3-2005
You can use .htaccess to change this link to:
www.mydomain.com/product.php?id=123
Within this script you can dynamically create each page with up-to-date information by querying your database based on the product id in this case 123.
Your script will provide each record with it's own title ('Nice BMW M3 2005'), a nice friendly URL ('www.mydomain.com/buy/123/nice-bmw-m3-2006') and you can include the correct meta information too, as well as images, reviews etc etc.
Job done, and you don't have to create hundreds of static HTML pages.
For more information on .htaccess, check out this tutorial.
What ILMV is trying to explain is that you have to have HTML pages that Google and other search engines can 'crawl' in order for them to index your content.
Since your information is loading dynamically from a database, you will need to use a server-side language like PHP to dynamically load information from the database and then output that information to an HTML page.
You have any number of options for how to accomplish this specifically, ILMV's suggestion is one of the better ways though.
Basically what you need to do first is figure out how to pull the information from the database and then use PHP (or another server-side language) to output the information to an HTML page.
Then you will need to determine whether you want to use the uglier, default url style for php driven pages:
mysite.com/products.php?id=123
But this url is not very user or search engine friendly and will result in your content not being indexed very well.
Or you can use some sort of URL rewriting mechanism (mod_rewrite in a .htaccess file does this or you can look at more complex PHP oriented solutions like Zend Framework that provide what's called a Front Controller to handle mapping of all requests) to make it so that your url's look like:
mysite.com/products/123/nice-bmw-m3-2006
This is what ILMV is talking about with regard to url masking.
Using this method of dynamically loading content will allow you to develop a single page to load the information for a number of different products based on the Id thus making it seem to the various search engines as though you have a unique page for each product.
Hello I've recently added a weather widget for people to add to their site.
Is basically a php page that people can link to.
My question is can I somehow put a counter or something to count how many people are using my widget?
my widget is located here http://www.site.com/widget/ it's in farsi language.
Well, it depends - do you want to know how many people have used your widget in their web site, or how many people have watched web sites that use your widgets? These numbers are completely different.
For the later option, jonstjohn's method would work great: For easy implementation and lots of features I recommend the Google Analytics way - they have really fancy graphs that show a lot of interesting information.
If on the other hand you want to count how many web sites are using your widget then you can do as follows:
Create a table in your database with a varchar column. Put a unique index on that column.
In your code, read the $_SERVER["HTTP_REFERER"] parameter to get the URL that called your widget.
Now you want to strip just the domain part from that URL as a web site would probably put your widget in their template so it will be available in all of their pages. So if you want to count web sites and not pages in web sites, then do something like
$domainParts = explode("/", $_SERVER["HTTP_REFERER"]);
$domain = $domainParts[2];
Now insert the domain you found out into your table. If this web site has already called your widget once then the insert would fail with a unique constraint error - just ignore the error (for example by using "#" on your insert command, like #mysql_query("INSERT INTO...") ).
To know how many web sites are using your widget, simply count the number of rows in the table.
You can place tracking code on the PHP page. Possibilities are:
Google analytics code
Record each time the .php page executes by inserting a row in a database
I'm sure there are others, but those should work.