I'm trying to get to the bottom of an issue Moz's crawler got stuck on. The easy problem we need to fix is that we have duplicates of the same page i.e.:
/capabilities/
/capabilities/index.html
That problem is occurring for a handful or directories. But we also have an issue that seems to be making an infinite loop of pages that can be accessed, just for this subdirectory:
/customer_service.html/
/customer_service.html/contact/index.html
/customer_service.html/contact_us/contact_form.php
/customer_service.html/contact/contact/contact/contact/index.html
/customer_service.html/contact/contact/contact_us/contact_form.php
/customer_service.html/contact_us/contact/contact/contact/index.html
/customer_service.html/contact_us/contact/contact/contact_us/contact_form.php
/customer_service.html/contact/contact_us/contact/contact_us/contact_us/contact/index.html
And on and on and on... I think it stopped crawling just because it reached 24,000 pages. All these pages actually work. Really there only need to be two pages: one for customer service FAQs, and one for contacting the company.
I'm a marketer, not a developer, so all I know is that this is an issue. I'm wondering whether we can fix this using htaccess, or if there is another problem. It seems to me like all these other pages need to be eliminated, not just redirected. Thanks.
edit: added more examples for illustrative and comic purposes
There are two things to do.
One is, like you say, not to allow these URLs to redirect to the main page. Show what you have in your .htaccess file, I will look how you could change it.
On the other hand, it is not sufficient to address the symptom. You have to heal the sickness. Here it means that you have some incorrect links on your site. Most probably these are relative URLs that are missing the initial slash ( contact instead of /contact ).
Related
I recently made some major changes to an ecommerce website that include url structure. The url to view a product is modified by .htaccess and contains a short product description that if changed will not affect the results on the page.
example: www.Example.com/staticFolder/non-deterministic-product-details/MODEL#.html
Now in the error log file I am seeing bingbot requesting pages like example.com/non-deterministic-product-details
Our sitemaps don't link to this page and I am not able to find any bad links on pages. Has anyone else had problems with bingbot doing this? I found another question that was locked for being random. Bingbot causing 404 errors. Is it more likely that I am doing something wrong? Should I avoid using psuedo directories in my .htaccess?
-Thanks
There's nothing requiring that spiders stick only to link-crawling. It's entirely possible it's guessing URLs which are similar to known ones in the hope that it'll find something.
At any rate, I wouldn't worry about it unless you know it's following a bad link. It's quite normal to get lots of requests for non-existent pages.
In my wordpress page, links are addressed in the way:
www.mywebsite.com/blog/?page_id=1234
And each number gives a different page. I would like to have a forwarding mechanism that would forward a page in the form
www.mywebsite.com/mypage/?page_id=1234
when done in the folder "mypage", to the other folder "blog" as:
www.mywebsite.com/blog/?page_id=1234
with transferring each variable I have on the URL (in this case the only variable is page_id, and I would like to have all the variables forwarded to another page).
I'm pretty experienced in HTML and C/C++, but a beginner in PHP (though I've noticed that PHP is very similar to c in its logic, and I've done many changes in my Wordpress page based on my knowledge in C).
How can I do such a forwarding of code? Is it possible?
If there's any information missing, please let me know.
Thank you for any efforts.
This website gave a good tutorial on this. It took me 15 minutes to implement the solution.
http://html.net/tutorials/php/lesson10.php
SESSION if you want it to be specific for every user, POST if you dont want to change the url and the same for every user.
Im not really sure I understood your question.
I have a problem.
I ma moving a system from one server to another and I came across a peculiar problem. There are some pages placed in a subfolder like these:
xttp://test.domain.com/admin/oders.php
xttp://test.domain.com/admin/users.php
xttp://test.domain.com/admin/whatever.php
Now, when I move around the pages, via some simple menu with links I get most of the times correct hits. But from time to time I end up on say:
xttp://test.domain.com/admin/admin/oders.php - which obviously causes 404
When I go back to previous page and press the link again it again works all right. Also when I hover over the links they always show proper paths regardless of whether I am going to get 404 or not. All links are dynamically generated by the scripts but they work perfectly on old server and as I say to a naked eye it all looks OK, right until I press the link.
Anyone has an idea where to look for a bug or which tool to use to see what is happening when I press the link? URL mod rewrite? Domain configuration? I am at a loss.
It sounds like the scripts are getting confused between
[xttp://test.domain.com]/admin/file.php
admin/file.php
file.php
Without seeing how the URLs are generated it's impossible to say how this is happening.
I have a rather complex problem (for me), which is why i need to turn to you guys :) Always heave been a great and fast ressource for help.
Background:
I have a website where the homepage is one big draggable screen (www.example.com/) where all the subpages are teasered. If you click on a Teaser (www.example.com/pages/page1) a jquery lightbox with iframe will open and present that page. That works great.
Problem:
Problem is, as always have been with iframes, if someone goes to the subpage directly (www.example.com/pages/page1), maybe via an indexed google search, he will not see the iframe and the big draggable screen behind it.
Question:
i'd need a script which, before anything happens, redirects an URL with more parts then root to the root page and append the parts as a different query, so i could read them out via javascript or something and trigger the iframelightbox per hand.
Approach:
so
www.example.com/pages/page1
needs to become
www.example.com?vars=/pages/page1
or something appropriate.
i think this is possible using mod_rewrite in my htaccess but i dont know how.
I hope this it understandable in some way :) i tried my best.
RewriteRule ^(.+)$ /?vars=$1
Should do the trick.
Don't put an * instead of + or you may end in infinite loop.
I'm not a very experienced programmer, and am using CodeIgniter for second time.
Suppose I have www.domain1.com. So I will have, say 3 controllers /area1, /area2, /area3. Users can access them as www.domain1.com/area1 etc. if I set the base URL as www.domain1.com. But my problem is, the client wants a certain area of the web, say area2, working as a microsite, in its own domain, so he wants to access area2 with www.domain2.com.
I don't know how to get this working with CodeIgniter. Suppose he registers www.domain2.com and set it pointing to the same DNS, server etc. How can I get CodeIgnitor to execute the controller area2 when the URL www.domain2.com is accessed?
Maybe changing $config['base-url']? Routing? .htaccess? Please, if you have solved this, examples of code involved would be greatly appreciated.
Edit: I will put example of the site I want to get.
I have one normal installation of CodeIgniter (external host, I can't access httpd.conf) It is on one machine, and the root of the site should be accessed by www.domain1.com
All domain are outside registered to. So I have the home controller, which shows me the main page view. And suppose the site have 3 areas /area1, /area2 /area3, with their correspondent controllers, showing these areas views.
My client want to emphasize one of the areas, the one that controller /area2 shows, and he want use a different domain for that area, www.domain2.com
What can I do so that when the user browse to www.domain2.com, CI redirects them to www.domain1.com/area2? Could I, for example, modify $config['base_url'] according the received URL, or is that impossible? Do I need to modify the .htaccess file?
After a lot of searching, I found a solution that seems to work, very easy to be honest:
Modify routes.php:
if ($_SERVER['HTTP_HOST']=="www.domain2.com") {
$route['default_controller'] = "area2";
}
No need for mod rewrite.
Here's a helpful link....
http://www.askaboutphp.com/88/codeigniter-setting-up-multiple-sites-on-one-install.html
Never actually done this myself, but this seems to be the way to go about it without having two ci installs. Good luck.
you definitely need to go the mod_rewrite way
the first solution that comes to my mind is to use Apache mod_rewrite, but as far as I know that would work only for internal redirects (i.e. resources residing on the same server/domain).
What about using an iframe? You could set up domain2.com home page with a full-page iframe that takes it's content from domain1.com/area2.