I create an scraper for an automoto site and first I want to get all manufactures and after that all links of models for each manufactures but with the code below I get only the first model on the list. Why?
<?php
$dom = new DOMDocument();
#$dom->loadHTMLFile('http://www.auto-types.com');
$xpath = new DOMXPath($dom);
$entries = $xpath->query("//li[#class='clearfix_center']/a/#href");
$output = array();
foreach($entries as $e) {
$dom2 = new DOMDocument();
#$dom2->loadHTMLFile('http://www.auto-types.com' . $e->textContent);
$xpath2 = new DOMXPath($dom2);
$data = array();
$data['newLinks'] = trim($xpath2->query("//div[#class='modelImage']/a/#href")->item(0)->textContent);
$output[] = $data;
}
echo '<pre>' . print_r($output, true) . '</pre>';
?>
SO I need to get: mercedes/100, mercedes/200, mercedes/300 but now with my script i get only the first link so mercedes/100...
please help
You need to iterate through the results instead of just taking the first item:
$items = $xpath2->query("//div[#class='modelImage']/a/#href");
$links = array();
foreach($items as $item) {
$links[] = $item->textContent;
}
$data['newLinks'] = implode(', ', $links);
Related
I've searched and tried multiple ways to get this but I'm not sure why it won't find most of the information on the webpage.
Page to scrape:
https://m.safeguardproperties.com/
Info needed:
Version number for PhotoDirect for Apple (currently 4.4.0)
Xpath to text needed (I think) : /html/body/div[1]/div[2]/div[1]/div[4]/div[3]/a
Attempts:
<?php
$file = "https://m.safeguardproperties.com/";
$doc = new DOMDocument();
$doc->loadHTMLFile($file);
$xpath = new DOMXpath($doc);
$elements = $xpath->query("/html/body/div[1]/div[2]/div[1]/div[4]/div[3]/a");
echo "<PRE>";
if (!is_null($elements)) {
foreach ($elements as $element) {
var_dump ($element);
echo "<br/>[". $element->nodeName. "]";
$nodes = $element->childNodes;
foreach ($nodes as $node) {
echo $node->nodeValue. "\n";
}
}
}
echo "</PRE>";
?>
Second Attempt:
<?PHP
$file = "https://m.safeguardproperties.com/";
$doc = new DOMDocument();
$doc->loadHTMLFile($file);
echo '<pre>';
// trying to find all links in document to see if I can see the correct one
$links = [];
$arr = $doc->getElementsByTagName("a");
foreach($arr as $item) {
$href = $item->getAttribute("href");
$text = trim(preg_replace("/[\r\n]+/", " ", $item->nodeValue));
$links[] = [
'href' => $href,
'text' => $text
];
}
var_dump($links);
echo '</pre>';
?>
For that particular website, the versions are being loaded from JSON data client side, you won't find them in the base document.
http://m.safeguardproperties.com/js/photodirect.json
This was located by comparing the original document source to the finished DOM and inspecting the network activity in the developer console.
$url = 'https://m.safeguardproperties.com/js/photodirect.json';
$json = file_get_contents( $url );
$object = json_decode( $json );
echo $object->ios->version; //4.4.0
Please respect other websites and cache your GET request.
Here is my code:
$res = file_get_contents("http://www.lenzor.com/photo/search/index/type/user/%D8%B9%D9%84%DB%8C//text/%D9%81%D8%A7%D8%B7%D9%85%D9%87");
$doc = new \DOMDocument();
#$doc->loadHTMLFile($res);
$xpath = new \DOMXpath($doc);
$links = $xpath->query("//ul[#class='user_box']/li");
$result = array();
if (!is_null($links)) {
foreach ($links as $link) {
$href = $link->getAttribute('class');
$result[] = [$href];
}
}
print_r($result);
Here is the content I'm working on. I mean it's the result of echo $res.
Ok well, the result of my code is an empty array. So $links is empty and that foreach won't be executed. Why? Why //ul[#class='user_box']/li query doesn't match the DOM ?
Expected result is an array contains the class attribute of lis.
Try this, Hope this will be helpful. There are few mistakes in your code.
1. You should search like this '//ul[#class="user_box clearfix"]/li' because class="user_box clearfix" class attribute of that HTML source contains two classes.
2. You should use loadHTMLinstead of loadHTMLFile.
<?php
ini_set('display_errors', 1);
libxml_use_internal_errors(true);
$res = file_get_contents("http://www.lenzor.com/photo/search/index/type/user/%D8%B9%D9%84%DB%8C//text/%D9%81%D8%A7%D8%B7%D9%85%D9%87");
$doc = new \DOMDocument();
$doc->loadHTML($res);
$xpath = new \DOMXpath($doc);
$links = $xpath->query('//ul[#class="user_box clearfix"]/li');
$result = array();
if (!is_null($links)) {
foreach ($links as $link) {
$href = $link->getAttribute('class');
$result[] = [$href];
}
}
print_r($result);
<?php
$i=1;
while ($i<=5) {
# code...
$url = 'http://www.amazon.in/gp/bestsellers/electronics/ref=zg_bs_nav_0#'.$i;
echo $url;
$html= file_get_contents($url);
$dom = new DOMDocument();
#$dom->loadHTML($html);
$xPath = new DOMXPath($dom);
$classname="zg_title";
$elements = $xPath->query("//*[contains(#class, '$classname')]");
foreach ($elements as $e)
{
$lnk = $e->getAttribute('href');
$e->setAttribute("href", "http://www.amazon.in".$lnk);
$newdoc = new DOMDocument;
$e = $newdoc->importNode($e, true);
$newdoc->appendChild($e);
$html = $newdoc->saveHTML();
echo $html;
}
$i++;
}
?>
I am trying to crawl through the Amazon bestsellers page which has a list of top 100 bestseller items which have 20 items in each page. In every loop the $i value is changed and appended to URL. But only the first 20 items are being displayed 5 times, I think this has something to do with the ajax pagination, but i am not able to figure out what it is.
Try this:
<?php
$i=1;
while ($i<=5) {
# code...
$url = 'http://www.amazon.in/gp/bestsellers/electronics/ref=zg_bs_electronics_pg_'.$i.'?ie=UTF8&pg='.$i;
echo $url;
$html= file_get_contents($url);
$dom = new DOMDocument();
#$dom->loadHTML($html);
$xPath = new DOMXPath($dom);
$classname="zg_title";
$elements = $xPath->query("//*[contains(#class, '$classname')]");
foreach ($elements as $e)
{
$lnk = $e->getAttribute('href');
$e->setAttribute("href", "http://www.amazon.in".$lnk);
$newdoc = new DOMDocument;
$e = $newdoc->importNode($e, true);
$newdoc->appendChild($e);
$html = $newdoc->saveHTML();
echo $html;
}
$i++;
}
?>
Change your $url
How do I rename a value in xml using PHP? This is what I've got so far:
<?php
$q = $_GET["q"];
$q = stripslashes($q);
$q = explode('|^', $q);
$old = $q[0];
$dom = new DOMDocument;
$dom->preserveWhiteSpace = false;
$dom->formatOutput = true;
$dom->Load("test.xml");
$xpath = new DOMXPath($dom);
$query1 = 'channel/item[title="' . $old . '"]/title';
$entries = $xpath->query($query1);
foreach ($entries as $entry)
{
$oldchapter = $entry->parentNode->removeChild($entry);
$item = $dom->getElementsByTagName('item');
foreach ($item as $items)
{
$title = $dom->createElement('title', $q[1]);
$items->appendChild($title);
}
}
$dom->save("test.xml");
Basically, what it does is take two titles from a url, the old existing title, and the one the user wants to change it to (so like this oldtitle|^newtitle), and puts them into an array.
What I've tried doing is removing the existing old title, and then making a new title with, using the new title value from the url, but it doesn't seem to be working. Where am I going wrong, or is there an easier way of doing this?
The way to do this is with DOMNode::replaceChild(). The majority of your code is correct, you've just slightly over-complicated some of the DOM stuff.
Try this:
<?php
$q = $_GET["q"];
$q = stripslashes($q);
$q = explode('|^', $q);
$old = $q[0];
$dom = new DOMDocument;
// Do this *before* loading the document
$dom->preserveWhiteSpace = false;
$dom->formatOutput = true;
$dom->Load("test.xml");
$xpath = new DOMXPath($dom);
$query1 = 'channel/item[title="' . $old . '"]/title';
$entries = $xpath->query($query1);
// This is all you need to do in the loop
foreach ($entries as $oldTitle) {
$newTitle = $dom->createElement('title', $q[1]);
$entry->parentNode->replaceChild($newTitle, $oldTitle);
}
$dom->save("test.xml");
Looked at a few other SO posts on this but no joy.
I've got this code:
$url = "http://itunes.apple.com/us/rss/toppaidapplications/limit=10/genre=6014/xml";
$string = file_get_contents($url);
$string = preg_replace("/(<\/?)(\w+):([^>]*>)/", "$1$2$3", $string);
$xml = simplexml_load_string($string);
foreach ($xml->entry as $val) {
echo "RESULTS: " . $val->attributes() . "\n";
but I can't get any results.
I'm specifically interested in getting the ID value which would be 549592189 in this fragment:
<id im:id="549592189" im:bundleId="com.activision.wipeout">http://itunes.apple.com/us/app/wipeout/id549592189?mt=8&uo=2</id>
Any suggestions?
SimpleXML gives you can easy way to drill down in the XML structure and get the element(s) you want. No need for the regex, whatever it does.
<?php
// Load XML
$url = "http://itunes.apple.com/us/rss/toppaidapplications/limit=10/genre=6014/xml";
$string = file_get_contents($url);
$xml = new SimpleXMLElement($string);
// Get the entries
$entries = $xml->entry;
foreach($entries as $e){
// Get each entriy's id
$id = $e->id;
// Get the attributes
// ID is in the "im" namespace
$attr = $id->attributes('im', TRUE);
// echo id
echo $attr['id'].'<br/>';
}
DEMO: http://codepad.viper-7.com/qNo7gs
Try with xpath:
$doc = new DOMDocument;
#$doc->loadHTML($string);
$xpath = new DOMXpath($doc);
$r = $xpath->query("//id/#im:id");
$id = $r->item(0)->value;
Try:
$sxml = new SimpleXMLElement($url);
for($i = 0;$i <=10;$i++){
$appid= $sxml->entry[$i]->id->attributes("im",TRUE);
echo $appid;
}