Ampersand issue in w3c validator and search engine - php

I'm using more than one ampersand in my url, see my link below
http://www.theonlytutorials.com/video.php?cat=55&vid=3975&auth=many
When i try to validate in w3c validator it showed hundreds of error because of this & (ampersand).
After that i read some post in here and i got the solution too.
Instead of using (&) If i use (&) w3c validates fine.
But the problem now is in search Engine. Instead of taking (&). it is taking like the below link
http://www.theonlytutorials.com/video.php?cat=55&vid=3975&auth=many
if you copy paste the above link in the address bar it will take you to the wrong page!. Please help how can i solve it.

There must be an error in your code but since we cannot see any of it I think the most important bit is to understand why the W3C validator complaints about raw &.
The HTML syntax contains two basic elements: tags (e.g. <strong>) and entities (e.g. €). Everything else is displayed as-is.
Browsers are expected to ignore errors.
When you type unknown or invalid tags, the browser will do its best to guess and fix it (you are probably aware of that already):
<p>Hello <i>world</b>!</p>
... will render as:
<p>Hello <i>world</i>!</p>
But the same happens when you type an unknown or invalid entity. In your example, there are two invalid entities:
http://www.theonlytutorials.com/video.php?cat=55&vid=3975&auth=many
^^^^ ^^^^^
However, it works because the browser is clever enough to figure out the real URL. Only the validator complaints because it is a tool specifically designed to find errors.
Now, imagine I want to use HTML to write an HTML tutorial and I want to explain the <strong> tag. If I just type <strong>example</strong>, the browser will display example. I need to encode the < symbol so it no longer has a special meaning:
<strong>example</strong>
Now the browser displays <strong>example</strong>, which is precisely the content I want to show.
The same happens with your URL. Since & is part of the entity syntax, when I want to insert a literal & I need to encode it as well:
Barnes & Noble
... will render as Barnes & Noble. Please note that this is only a syntactic trick to insert plain text into a HTML document. Your document shows Barnes & Noble. to all effects, no matter how you encode it. So when you replace & with & in your URL, you are not changing your URL, you are just encoding it.
If search engines are spidering the wrong URL, that means you have actually changed your URL rather than just encoding it, so the source code is:
http://www.theonlytutorials.com/video.php?cat=55&amp;vid=3975&amp;auth=many
... and renders as:
http://www.theonlytutorials.com/video.php?cat=55&vid=3975&auth=many
This can happen, for instance, if you encode twice:
<?php
$url = 'http://www.theonlytutorials.com/video.php?cat=55&vid=3975&auth=many';
$url = htmlspecialchars($url);
$url = htmlspecialchars($url);
echo $url;
... or:
<?php
$url = 'http://www.theonlytutorials.com/video.php?cat=55&vid=3975&auth=many';
$url = htmlspecialchars($url); // Oops: URL is already encoded!
echo $url;

Seems that you made a typo error, it must be & not &ampamp;

Related

Settings that could influence PHP str_replace behaviour

I am currently working on a replacement tool that will dynamically replace certain strings (including html) in a website using a smarty outputfilter.
For the replacement to take place, I am using PHP's str_ireplace method, which reads the code that is supposed to be replaced and the replacement code from a database, and then pass the result to the smarty output (using an output filter), in a similar way as the below.
$tpl_source = str_ireplace($replacements['sourceHTML'], $replacements['replacementHTML'], $tpl_source);
The problem is, that although it works great on my dev server, once uploaded to the live server replacements occasionally fail. The same replacements work just fine on my dev version though. After some examinations and googling there was not much I could find out regarding this issue. So my question is, what could influence str_replace's behavour?
Thanks
Edit with replacement example:
$htmlsource = file_get_contents('somefile.html');
$newstr = str_replace('Some text', 'sometext', $htmlsource); // the text to be replaced does exist in the html source
fails to replace. After some checking, it looks like the combination of "> creates a problem. But just the combination of it. If I try to change only (") it works, if I try to change only (>) it works.
It might be that special chars like umlauts do not display on the live server correctly and so str_replace() would fail, if there are specialchars inside the string you want to replace.
Is the input string identical on both systems? Have you verified this? Are you sure?
Things to check:
Are the HTML attributes in the same order?
Are the attribute values using the same kind quote marks? (eg <a href='#'> vs <a href="#">)
Is there any other stray HTML code getting in there?
Is the entity encoding the same? (eg vs   - same character; different HTML)
Is the character-set the same? (eg utf-8 vs ISO 8859-1: Accented characters will be encoded differently)
Any of these things will affect the result and produce the failures you're describing.
This was a trikcy problem, and it ended up having nothing to do with the str_replace method itself;
We are using smarty as a tamplating system. The str_replace method was used by a smarty ouput filter in order to change the html in some ocassions, just before it was delivered to the user.
Here is the Smarty outputfilter Code:
function smarty_outputfilter_replace($tpl_source, &$smarty)
{
$replacements = Content::getReplacementsForPage();
if (is_array($replacements))
{
foreach ($replacements as $replacementData)
{
$tpl_source = str_replace($replacementData['sourcecode'], $replacementData['replacementcode'], $tpl_source);
}
}
return ($tpl_source);
}
So this code failed now and then for now apparent reason... until I realized that the HTML code in the smarty template was being manipulated by an Apache filter.
This resulted into the source code in the browser (which we were using as the code to be replaced by something else) not being identical to the template code (which smarty was trying to modify). Result? str_replace failed :)

Ampersand problem in XML when creating a URL String

I am working with an XML feed that has, as one of it's nodes, a URL string similar to the following:
http://aflite.co.uk/track/?aid=13414&mid=32532&dl=http://www.google.com/&aref=chris
I understand that ampersands cause a lot of problems in XML and should be escaped by using & instead of a naked &. I therefore changed the php to read as follows:
<node><?php echo ('http://aflite.co.uk/track/?aid=13414&mid=32532&dl=http://www.google.com/&aref=chris'); ?></node>
However when this generates the XML feed, the string appears with the full &
and so the actual URL does not work. Apologies if this is a very basic misunderstanding but some guidance would be great.
I've also tried using %26 instead of & but still getting the same problem.
If you are inserting something into XML/HTML you should always use the htmlspecialchars function. this will escape your strings into correct XML syntax.
but you are running into a second problem.
your have added a second url to the first one.
this need also escaped into url syntax.
for this you need to use urlencode.
<node><?php echo htmlspecialchars('http://aflite.co.uk/track/?aid=13414&mid=32532&aref=chris&dl='.urlencode('http://www.google.com/')); ?></node>
& is correct for escaping ampersands in an XML document. The example you've given should work.
You state that it doesn't work, but you haven't stated what application you're using, or in what way it doesn't work. What exactly happens when you click the link? Do the & strings end up in the browser's URL field? If that's the case, it sounds like a fault with the software you've viewing the XML with. Have you tried looking at the XML in another application to see if the problem is consistent?
To answer the final part of your question: %26 would definitely not work for you -- this would be what you'd use if your URL parameters needed to contain ampersands. Say for example in aref=chris, if the name chris were to an ampersand (lets say the username was chris&bob), then that ampersand would need to be escaped using %26 so that the URL parser didn't see it as starting a new URL parameter.
Hope that helps.

Why is rawurlencode() in PHP adding additional escape characters to ampersands?

I think I'm missing something obvious here but it is driving me crazy and I can't figure it out. I'm developing a WordPress plugin and part of it needs to take the WordPress post title and send that to a RESTful web service to do something else. So of course I want to rawurlencode() the post title since who knows what text might be in there. However, for some reason the output I'm getting has extra escape characters and I have no idea where they are coming from (and it's causing problems with the web service I'm calling obviously).
My code is fairly straight forward:
$topic = get_the_title($post_id);
$curl_post_fields = 'name=' . rawurlencode( $topic );
Yet when I print the output of those two strings I get:
topic=a & b
name=a%20%26%23038%3B%20b
Whereas I would expect the URL encoded string to be
name=a%20%26%20b
I have no idea where that extra %23038%3B could be coming from. If I'm reading the encoding on that correctly it translates to #038; but I still don't know where it's coming from.
There seems to be a html encoding in between as well, instead of &, & is in the encoded string. Probably because & has to be escaped in html, and the get_title function escapes this using html_special_chars or something like that.
I had some problems with that when i used an older php version

XML parser error: entity not defined

I have searched stackoverflow on this problem and did find a few topics, but I feel like there isn't really a solid answer for me on this.
I have a form that users submit and the field's value is stored in a XML file. The XML is set to be encoded with UTF-8.
Every now and then a user will copy/paste text from somewhere and that's when I get the "entity not defined error".
I realize XML only supports a select few entities and anything beyond that is not recognized - hence the parser error.
From what I gather, there's a few options I've seen:
I can find and replace all and swap them out with or an actual space.
I can place the code in question within a CDATA section.
I can include these entities within the XML file.
What I'm doing with the XML file is that the user can enter content into a form, it gets stored in a XML file, and that content then gets displayed as XHTML on a Web page (parsed with SimpleXML).
Of the three options, or any other option(s) I'm not aware of, what's really the best way to deal with these entities?
Thanks,
Ryan
UPDATE
I want to thank everyone for the great feedback. I actually determined what caused my entity errors. All the suggestions made me look into it more deeply!
Some textboxes where plain old textboxes, but my textareas were enhanced with TinyMCE. It turns out, while taking a closer look, that the PHP warnings always referenced data from the TinyMCE enhanced textareas. Later I noticed on a PC that all the characters were taken out (because it couldn't read them), but on a MAC you could see little square boxes referencing the unicode number of that character. The reason it showed up in squares on a MAC in the first place, is because I used utf8_encode to encode data that wasn't in UTF to prevent other parsing errors (which is somehow also related to TinyMCE).
The solution to all this was quite simple:
I added this line entity_encoding : "utf-8" in my tinyMCE.init. Now, all the characters show up the way they are supposed to.
I guess the only thing I don't understand is why the characters still show up when placed in textboxes, because nothing converts them to UTF, but with TinyMCE it was a problem.
I agree that it is purely an encoding issue. In PHP, this is how I solved this problem:
Before passing the html-fragment to SimpleXMLElement constructor I decoded it by using html_entity_decode.
Then further encoded it using utf8_encode().
$headerDoc = '<temp>' . utf8_encode(html_entity_decode($headerFragment)) . '</temp>';
$xmlHeader = new SimpleXMLElement($headerDoc);
Now the above code does not throw any undefined entity errors.
You could HTML-parse the text and have it re-escaped with the respective numeric entities only (like: →  ). In any case — simply using un-sanitized user input is a bad idea.
All of the numeric entities are allowed in XML, only the named ones known from HTML do not work (with the exception of &, ", <, >, &apos;).
Most of the time though, you can just write the actual character (ö → ö) to the XML file so there is no need to use an entity reference at all. If you are using a DOM API to manipulate your XML (and you should!) this is your safest bet.
Finally (this is the lazy developer solution) you could build a broken XML file (i.e. not well-formed, with entity errors) and just pass it through tidy for the necessary fix-ups. This may work or may fail depending on just how broken the whole thing is. In my experience, tidy is pretty smart, though, and lets you get away with a lot.
1. I can find and replace all [ ?] and swap them out with [ ?] or an actual space.
This is a robust method, but it requires you to have a table of all the HTML entities (I assume the pasted input is coming from HTML) and to parse the pasted text for entity references.
2. I can place the code in question within a CDATA section.
In other words disable parsing for the whole section? Then you would have to parse it some other way. Could work.
3. I can include these entities within the XML file.
You mean include the entity definitions? I think this is an easy and robust way, if you don't mind making the XML file quite a bit bigger. You could have an "included" file (find one on the web) which is an external entity, which you reference from the top of your main XML file.
One downside is that the XML parser you use has to be one that processes external entities (which not all parsers are required to do). And it must correctly resolve the (possibly relative) URL of the external entity to something accessible. This is not too bad but it may increase constraints on your processing tools.
4. You could forbid non-XML in the pasted content. Among other things, this would disallow entity references that are not predefined in XML (the 5 that Tomalak mentioned) or defined in the content itself. However this may violate the requirements of the application, if users need to be able to paste HTML in there.
5. You could parse the pasted content as HTML into a DOM tree by setting someDiv.innerHTML = thePastedContent;
In other words, create a div somewhere (probably display=none, except for debugging). Say you then have a javascript variable myDiv that holds this div element, and another variable myField that holds the element that is your input text field. Then in javascript you do
myDiv.innerHTML = myField.value;
which takes the unparsed text from myField, parses it into an HTML DOM tree, and sticks it into myDiv as HTML content.
Then you would use some browser-based method for serializing (= "de-parsing") the DOM tree back into XML. See for example this question. Then you send the result to the server as XML.
Whether you want to do this fix in the browser or on the server (as #Hannes suggested) will depend on the size of the data, how quick the response has to be, how beefy your server is, and whether you care about hackers sending not-well-formed XML on purpose.
Use "htmlentities()" with flag "ENT_XML1": htmlentities($value, ENT_XML1);
If you use "SimpleXMLElement" class:
$SimpleXMLElement->addChild($name, htmlentities($value, ENT_XML1));
If you want to convert all characters, this may help you (I wrote it a while back) :
http://www.lautr.com/convert-all-applicable-characters-to-numeric-entities-for-use-in-xml
function _convertAlphaEntitysToNumericEntitys($entity) {
return '&#'.ord(html_entity_decode($entity[0])).';';
}
$content = preg_replace_callback(
'/&([\w\d]+);/i',
'_convertAlphaEntitysToNumericEntitys',
$content);
function _convertAsciOver127toNumericEntitys($entity) {
if(($asciCode = ord($entity[0])) > 127)
return '&#'.$asciCode.';';
else
return $entity[0];
}
$content = preg_replace_callback(
'/[^\w\d ]/i',
'_convertAsciOver127toNumericEntitys', $content);
This question is a general problem for any language that parses XML or JSON (so, basically, every language).
The above answers are for PHP, but a Perl solution would be as easy as...
my $excluderegex =
'^\n\x20-\x20' . # Don't Encode Spaces
'\x30-\x39' . # Don't Encode Numbers
'\x41-\x5a' . # Don't Encode Capitalized Letters
'\x61-\x7a' ; # Don't Encode Lowercase Letters
# in case anything is already encoded
$value = HTML::Entities::decode_entities($value);
# encode properly to numeric
$value = HTML::Entities::encode_numeric($value, $excluderegex);

Why mysql is not storing data after "#" character?

I have made one form in which there is rich text editor. and i m trying to store the data to database.
now i have mainly two problem..
1) As soon as the string which contents "#"(basically when i try to change the color of the font) character, then it does not store characters after "#". and it also not store "#" character also.
2) although i had tried....in javascript
html.replace("\"","'");
but it does not replace the double quotes to single quotes.
We'll need to see some code. My feeling is you're missing some essential escaping step somewhere. In particular:
As soon as the string which contents "#"(basically when i try to change the color of the font) character
Implies to me that you might be sticking strings together into a URL like this:
var url= '/something.php?content='+html;
Naturally if the html contains a # symbol, you've got problems, because in:
http://www.example.com/something.php?content=<div style="color:#123456">
the # begins a fragment identifier called #123456">, like when you put #section on the end of a URL to go to the anchor called section in the HTML file. Fragment identifiers are purely client-side and are not sent to the server, which would see:
http://www.example.com/something.php?content=<div style="color:
However this is far from the only problem with the above. Space, < and = are simly invalid in URLs, and other characters like & will also mess up parameter parsing. To encode an arbitrary string into a query parameter you must use encodeURIComponent:
var url= '/something.php?content='+encodeURIComponent(html);
which will replace # with %35 and similarly for the other out-of-band characters.
However if this is indeed what you're doing, you should in any case you should not be storing anything to the database in response to a GET request, nor relying on a GET to pass potentially-large content. Use a POST request instead.
It seems that you are doing something very strange with your database code. Can you show the actual code you use for storing the string to database?
# - character is a common way to create a comment. That is everything starting from # to end of line is discarded. However if your code to store to database is correct, that should not matter.
Javascript is not the correct place to handle quote character conversions. The right place for that is on server side.
As you have requested....
I try to replay you... I try to mention exact what I had done...
1) on the client side on the html form page I had written like this..
html = html.trim(); // in html, the data of the rich text editor will come.
document.RTEDemo.action = "submit.php?method='"+ html.replace("\"","'") + "'";
\\ i had done replace bcz i think that was some problem with double quotes.
now on submit.php , my browser url is like this...
http://localhost/nc/submit.php?method='This is very simple recipe.<br><strong style='background-color: #111111; color: #80ff00; font-size: 20px;">To make Bread Buttor you will need</strong><br><br><blockquote><ol><li>bread</li><li>buttor</li></ol></li></blockquote><span style="background-color: #00ff80;">GOOD.</span><br><br><br><blockquote><br></blockquote><br>'
2) on submit.php ........I just write simply this
echo "METHOD : ".$_GET['method'] . "<br><br>";
$method = $_GET['method'];
now my answer of upper part is like this...
METHOD : 'This is very simple recipe.
now i want to store the full detail of URL....but its only storing...
This is very simple recipe.

Categories