External entities not working in simplexml - php

I am parsing XML using SimpleXML in PHP 5 and external entities are not working. The XML parses, but the entities just are blank. The underlying library is libxml2.
This is the code:
libxml_disable_entity_loader(false);
simplexml_load_file($target_file);
It parses the XML as expected, but doesn't resolve the external entities and seems to ignore them.

This is the expected behavior, because you need to tell when loading the document to expand (and therefore remove) these entities:
libxml_disable_entity_loader(false);
simplexml_load_file($target_file, 'SimpleXMLElement', LIBXML_NOENT);
############
This constant is also cross-linked on the manual page of libxml_disable_entity_loader.
The function itself only enables or disables the default entity loader. Additionally the parser needs to be told via the libxml2 based option flag, that those entities should be substituted. Only then the default loader (or if you have set it to a different one) would kick in.
Online Demo:
<?php
/**
* #link https://stackoverflow.com/a/29864193/367456
*/
$buffer = <<<XML
<?xml version="1.0"?>
<!DOCTYPE foo [
<!ELEMENT foo ANY >
<!ENTITY xxe SYSTEM "data://text/plain,test" >]><foo>&xxe;</foo>
XML;
libxml_disable_entity_loader(false);
$xml = simplexml_load_string($buffer);
$xml->asXML('php://output');
$xml = simplexml_load_string($buffer, 'SimpleXMLElement', LIBXML_NOENT);
$xml->asXML('php://output');
Output for 5.2.11 - 5.6.8, php7#20140507 - 20150401, hhvm-3.5.0 - 3.6.1
<?xml version="1.0"?>
<!DOCTYPE foo [
<!ELEMENT foo ANY>
<!ENTITY xxe SYSTEM "data://text/plain,test">
]>
<foo>&xxe;</foo>
<?xml version="1.0"?>
<!DOCTYPE foo [
<!ELEMENT foo ANY>
<!ENTITY xxe SYSTEM "data://text/plain,test">
]>
<foo>test</foo>
The XML is based on the exploit examples outlined on XML External Entity (XXE) Processing (OWASP Wiki) and modified for the PHP demonstration regarding your question.
And more important than PHP versions is the version of libxml both on system and the binding in PHP. Just saying in case the 3v4l.org demo code creates the impression that it always behaved the same in all PHP versions - this just must not be the case.
Related Q&A
Clarifications on XXE vulnerabilities throughout PHP versions (Jun 2014)
Check for malicious XML before allowing DTD loading? (Jul 2014)

Related

Libxml_disable_entitiy_loader(false); Still no external entities loaded

I've been trying to mimic the XXE vulnerability on my virtual machine for learning purposes. However, I think I misunderstand something. When I call libxml_disable_external_entities(false); I would think that external entities are now loaded, however, external entities does not seem to load.
When I set the flag LIBXML_NOENT, it does work. But that leaves me with the question, what does libxml_disable_external_entities actually do then? I have set it both to false and true, with the same results.
So I went to the internet with the same question and noticed a CVE report on the ubuntu website (I have tested on ubuntu). The CVE stated that libxml is standard vulnerable to XXE. The fix was to stop loading external entities by default. See: https://usn.ubuntu.com/1904-1/
My test setup:
<?php
$xml= '<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE foo [
<!ELEMENT foo ANY >
<!ENTITY xxe SYSTEM "http://www.localhost:8000" >]><foo>&xxe;</foo>';
libxml_disable_entity_loader(false);
$document = new DOMDocument();
$document->loadXML($xml);
?>
I expected my webserver to receive a call, but it did not. The PHP handler gives me no output. So my question: what does libxml_disable_entity_loader actually do?
PS: Could not find anything in libxml code either. Could find the fix to stop loading external entities unless otherwise stated, though.
Thanks!
UPDATE: I have found that it only works with the combination if libxml_disable_entitiy_loader(false); and the LIBXML_NOENT flag set in the 'loadXML' method.
EDIT:
The method does the following (in PHP code):
PHP_LIBXML_API zend_bool php_libxml_disable_entity_loader(zend_bool disable) /* {{{ */
{
zend_bool old = LIBXML(entity_loader_disabled);
LIBXML(entity_loader_disabled) = disable;
return old;
}
However, searching for entity_loader_disabled yields no results in libxml code.

Tell XMLReader to ignore provided namespace info

I'm using an instance of PHPs built-in XMLReader to read some kind of user-generated XML file. Usually this XML files content starts like the following sample snippet, where everything works fine:
<?xml version="1.0" encoding="UTF-8"?>
<openimmo>
<uebertragung art="OFFLINE" umfang="VOLL" version="1.2.7" (...)
However, another user uses a different software to send and generate the XML file. The XML generated by this software starts like:
<?xml version="1.0" encoding="UTF-8"?>
<openimmo xmlns="http://www.openimmo.de" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openimmo.de openimmo.xsd">
<uebertragung art="OFFLINE" umfang="VOLL" version="1.2.7" (...)
Which causes my importer to fail with the following error:
XMLReader::read(): Element '{http://www.openimmo.de}openimmo': No matching global declaration available for the validation root.
I'm already doing validation by manually applying some XSD schema. The passed file follows the same schema, just explicitly specifies the xmlns attributes. How can I work around this issue? How can I tell XMLReader to just ignore that xmlns statement?
My code (simplified to the relevant sections) looks like the following snippet:
$reader = new XMLReader();
$success = #$reader->open($path);
if (!$success) { /* error handling */ }
$reader->setSchema($localOpenImmoXsdPath);
/* then starts reading and throws the above exception */
Namespace information is fundamental and there's no way an XML parser is going to ignore it.
Your options are either (a) send the file back to sender, saying it doesn't conform to the agreed schema, or (b) transform the file sent to you so that it does conform, by changing the namespace. That's a fairly simple XSLT transformation.
My immediate instinct was to look at the OpenImmo specs to see what they say about namespaces and schema conformance, but unfortunately access to the specs requires registration and licensing. Basically, either the specs allow both these formats, which would be a pretty shoddy spec, or they only allow one of them, in which case you shouldn't be accepting both.

Clarifications on XXE vulnerabilities throughout PHP versions

I post a question here as a last resort, I have browsed the web and went through many attempts but did not succeed.
Replicating a XXE attack is what I am trying to do, in order to prevent them, but I cannot seem to get my head around the way PHP works with XML entities. For the record I am using PHP 5.5.10 on Ubuntu 12.04, but I have done some tests on 5.4 and 5.3, and libxml2 seem to be of version 2.7.8 (which does not seem to include the default to not resolving entities).
In the following example, calling libxml_disable_entity_loader() with true or false has no effect, or I am doing something wrong.
$xml = <<<XML
<?xml version="1.0"?>
<!DOCTYPE root [
<!ENTITY c PUBLIC "bar" "/etc/passwd">
]>
<root>
<test>Test</test>
<sub>&c;</sub>
</root>
XML;
libxml_disable_entity_loader(true);
$dom = new DOMDocument();
$dom->loadXML($xml);
// Prints Test.
print $dom->textContent;
But, I could specifically pass some arguments to loadXML() to allow some options, and that works when the entity is a local file, not when it is an external URL.
$xml = <<<XML
<?xml version="1.0"?>
<!DOCTYPE root [
<!ENTITY c PUBLIC "bar" "/etc/passwd">
]>
<root>
<test>Test</test>
<sub>&c;</sub>
</root>
XML;
$dom = new DOMDocument();
$dom->loadXML($xml, LIBXML_NOENT | LIBXML_DTDLOAD);
// Prints Test.
print $dom->textContent;
Now if we are changing the entity to something else, as in the following example, the entity is resolved but I could not disable it at all using the parameters or function... What is happening?!
$xml = <<<XML
<?xml version="1.0"?>
<!DOCTYPE root [
<!ENTITY c "Blah blah">
]>
<root>
<test>Test</test>
<sub>&c;</sub>
</root>
XML;
$dom = new DOMDocument();
$dom->loadXML($xml);
// Prints Test.
print $dom->textContent;
The only way that I could find was to overwrite the properties of the DOMDocument object.
resolveExternals set to 1
substituteEntities set to 1
Then they are resolved, or not.
So to summarise, I would really like to understand what I am obviously not understanding . Why do those parameters and function seem to have no effect? Is libxml2 taking precedence over PHP?
Many thanks!
References:
https://www.owasp.org/index.php/XML_External_Entity_%28XXE%29_Processing
http://au2.php.net/libxml_disable_entity_loader
http://au2.php.net/manual/en/libxml.constants.php
http://www.vsecurity.com/download/papers/XMLDTDEntityAttacks.pdf
http://www.mediawiki.org/wiki/XML_External_Entity_Processing
How can I use PHP's various XML libraries to get DOM-like functionality and avoid DoS vulnerabilities, like Billion Laughs or Quadratic Blowup?
Keeping it simple .. As it should be simple :-)
Your first code snippet
libxml_disable_entity_loader does or does not do anything here based on whether your system resolves entities by default or not (mine does not). This is controlled by LIBXML_NOENT option of libxml.
Without it the document processor may not even try translating external entities and therefore libxml_disable_entity_loader has nothing to really influence (if libxml does not load entities by default which seems to be the case in your test-case).
Add LIBXML_NOENT to loadXML() like this:
$dom->loadXML($xml, LIBXML_NOENT);
and you'll quickly get:
PHP Warning: DOMDocument::loadXML(): I/O warning : failed to load external entity "/etc/passwd" in ...
PHP Warning: DOMDocument::loadXML(): Failure to process entity c in Entity, line: 7 in ...
PHP Warning: DOMDocument::loadXML(): Entity 'c' not defined in Entity, line: 7 in ...
Your second code snippet
In this scenario you've enabled entity resolving by using the LIBXML_NOENT option, that's why it goes after /etc/passwd.
The example works just fine on my machine even for external URL - I changed the ENTITY to an external one like this:
<!ENTITY c PUBLIC "bar" "https://stackoverflow.com/opensearch.xml">
It can, however, be even influenced by eg. allow_url_fopen PHP INI setting - put it to false and PHP won't ever load a remote file.
Your third code snippet
XML Entity that you've provided is not an external one but rather an internal one (see eg. here).
Your entity:
<!ENTITY c "Blah blah">
How internal entity is defined:
<!ENTITY % name "entity_value">
Therefore there is no reason for PHP or libxml to prevent resolving such entity.
Conclusion
I've quickly put up a PHP XXE tester script which tries out different settings and shows whether XXE is successful and in which case.
The only line that should actually show a warning is the "LIBXML_NOENT" one.
If any other line loads the WARNING, external entity loaded! your setup does allow loading external entities by default.
You can't go wrong by using SHOULD USE libxml_disable_entity_loader() regardless of your/your provider's machine default settings. If your app ever gets migrated it might become vulnerable instantly.
correct usage
As the MediaWiki states in link you've posted.
Unfortunately, the way that libxml2 implements the disabling, the library is crippled when external entities are disabled, and functions that would otherwise be safe cause an exception in the entire parsing.
$oldValue = libxml_disable_entity_loader(true);
// do whatever XML-processing related
libxml_disable_entity_loader($oldValue);
Note: libxml_disable_entity_loader() also prohibits loading external xml files directly (not through entities):
<?php
$remote_xml = "https://stackoverflow.com/opensearch.xml";
$dom = new DOMDocument();
if ($dom->load($remote_xml) !== FALSE)
echo "loaded remote xml!\n";
else
echo "failed to load remote xml!\n";
libxml_disable_entity_loader(true);
if ($dom->load($remote_xml) !== FALSE)
echo "loaded remote xml after libxml_disable_entity_loader(true)!\n";
else
echo "failed to remote xml after libxml_disable_entity_loader(true)!\n";
On my machine:
loaded remote xml!
PHP Warning: DOMDocument::load(): I/O warning : failed to load external entity "https://stackoverflow.com/opensearch.xml" in ...
failed to remote xml after libxml_disable_entity_loader(true)!
It might perhaps be related to this PHP bug but PHP is being really stupid about it as:
libxml_disable_entity_loader(true);
$dom->loadXML(file_get_contents($remote_xml));
works just fine.

SimpleXML parent - child issue

i am having an issue with parsing an XML file using SimpleXML and PHP.
The XML file in question is provided by a third party and includes a number of child elements (going down multiple levels) within it. I know which elements i require and can see them within the XML file, but i just can't seem to get them to print using PHP.
Example XML feed for test.xml:
<?xml version="1.0" encoding="utf-8"?>
<Element1 xmlns="" release="8.1" environment="Production" lang="en-US">
<Element2>
<Element3>
<Element4>
<Element5>it worked</Element5>
</Element4>
</Element3>
</Element2>
</Element1>
The file only includes one of each attribute so i can be very particular with the request, the code i have so far is below:
$lib=simplexml_load_file("test.xml");
$make=$lib->Element1->Element2->Element3->Element4->Element5;
print $make;
I have tried to look this up before asking, but the only solutions i can see are when the child attributes are unknown or there are multiple results for each request, which is not the case in this instance.
Any help or guidance would be greatly received.
Thanks
In your code above, $lib is Element1. So you just need to drop one of your references. This:
$make=$lib->Element1->Element2->Element3->Element4->Element5;
Should become this:
$make=$lib->Element2->Element3->Element4->Element5;
Also, SimpleXML is an awful awful awful awful interface (considering that "Simple" is in the name and there is mass confusion about how to use it). I would always recommend DOMDocument instead.
I'd strongly recommend using xpath as it will give you more flexibility e.g. Allow you to restrict results based on xml node attributes.
$xml = simplexml_load_string('<?xml version="1.0" encoding="utf-8"?>
<Element1 xmlns="" release="8.1" environment="Production" lang="en-US">
<Element2>
<Element3>
<Element4>
<Element5>it worked</Element5>
</Element4>
</Element3>
</Element2>
</Element1>');
$data=$xml->xpath('/Element1/Element2/Element3/Element4/Element5');
echo (string)$data[0]; //outputs 'it worked'
//this also works
$data=$xml->xpath('//Element5');
echo (string)$data[0]; //outputs 'it worked'

XML validation against given DTD in PHP

In PHP, I am trying to validate an XML document using a DTD specified by my application - not by the externally fetched XML document. The validate method in the DOMDocument class seems to only validate using the DTD specified by the XML document itself, so this will not work.
Can this be done, and how, or do I have to translate my DTD to an XML schema so I can use the schemaValidate method?
(this seems to have been asked in Validate XML using a custom DTD in PHP but without correct answer, since the solution only relies on DTD speicified by the target XML)
Note: XML validation could be subject to the Billion Laughs attack, and similar DoS vectors.
This essentially does what rojoca mentioned in his comment:
<?php
$xml = <<<END
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE foo SYSTEM "foo.dtd">
<foo>
<bar>baz</bar>
</foo>
END;
$root = 'foo';
$old = new DOMDocument;
$old->loadXML($xml);
$creator = new DOMImplementation;
$doctype = $creator->createDocumentType($root, null, 'bar.dtd');
$new = $creator->createDocument(null, null, $doctype);
$new->encoding = "utf-8";
$oldNode = $old->getElementsByTagName($root)->item(0);
$newNode = $new->importNode($oldNode, true);
$new->appendChild($newNode);
$new->validate();
?>
This will validate the document against the bar.dtd.
You can't just call $new->loadXML(), because that would just set the DTD to the original, and the doctype property of a DOMDocument object is read-only, so you have to copy the root node (with everything in it) to a new DOM document.
I only just had a go with this myself, so I'm not entirely sure if this covers everything, but it definitely works for the XML in my example.
Of course, the quick-and-dirty solution would be to first get the XML as a string, search and replace the original DTD by your own DTD and then load it.
I think that's only possible with XSD, see:
http://php.net/manual/en/domdocument.schemavalidate#62032

Categories