The following snippet of PHP code creates $desc alright, but I like it to introduce two (2) blank spaces between every dpItemFeatureList found as it goes through its iteration.
I can't seem to garner exactly what or where to add a snippet to do this?
function get_description($asin){
$url = 'http://www.amazon.com/gp/aw/d/' . $asin . '?d=f&pd=1';
$data = request_data($url);
$desc = '';
if ($data) {
$dom = new DOMDocument();
#$dom->loadHTML($data);
$xpath = new DOMXPath($dom);
if (preg_match('#dpItemFeaturesList#',$data)){
$k = $xpath->query('//ul[#class="dpItemFeaturesList"]');
foreach ($k as $c => $tot) {
$desc .= $tot->nodeValue;
}
}
}
return $desc;
Looking at the code you have shared here and consequently having a look at the data that you are processing (a sample of which I have pasted here) you actually want to collect the text within the <li> child elements of the <ul class="dpItemFeaturesList"> node.
In your original code snippet your XPath is as follows:
'//ul[#class="dpItemFeaturesList"]'
This will only select the <ul> element and not the child elements. Consequently when you try to do a $tot->nodeValue it will concatenate all the text within all it's child nodes without spaces (ah ha, the real reason why you want spaces in the first place).
To fix this we should do two things:
Select the <li> nodes within the appropriate node. Change the XPath to //ul[#class="dpItemFeaturesList"]/li.
In the foreach loop concatenate 2 non-breakable spaces (because this is HTML) to the $desc variable.
Here $c is the array index.
function get_description($asin){
$url = 'http://www.amazon.com/gp/aw/d/' . $asin . '?d=f&pd=1';
$data = request_data($url);
$desc = '';
if ($data) {
$dom = new DOMDocument();
#$dom->loadHTML($data);
$xpath = new DOMXPath($dom);
if (preg_match('#dpItemFeaturesList#',$data)){
$k = $xpath->query('//ul[#class="dpItemFeaturesList"]/li');
foreach ($k as $c => $tot) {
if ($c > 0) {
$desc .= " ";
}
$desc .= $tot->nodeValue;
}
}
}
return $desc;
}
We check for $c > 0 so that you will not get extra spaces after the last node in the loop.
P.S.: Unrelated to your original question. The code for which you shared a link has an undefined variable $timestamp in $date = date("format", $timestamp); on line 116.
Since you're appending everything to desc, try something like
$desc .= $tot->nodeValue;
$desc .= "<br />"
try that:
$desc .= $tot->nodeValue.' ';
and trim($desc) after the loop to avoid two spaces at the end.
or, alternatively create an array:
$desc = array();
//....
$desc[] = $tot->nodeValue;
and return implode(' ', $desc)
If you need that between each one, you need to add in front on each iteration but the first:
$k = $xpath->query('//ul[#class="dpItemFeaturesList"]');
foreach ($k as $c => $tot) {
$c && $desc .= ' '; # all but first
$desc .= $tot->nodeValue;
}
This is an expression which saves you an if but it works similar. Maybe a bit of taste so sure, an if can do it as well:
$k = $xpath->query('//ul[#class="dpItemFeaturesList"]');
foreach ($k as $c => $tot) {
if($c) $desc .= ' '; # all but first
$desc .= $tot->nodeValue;
}
This works because every integer number but zero is true in PHP.
See the demo.
Related
I'm getting some data from a webpage for clients and that works fine, it gets all data in seperate rows by exploding the \n into new lines which I then map to specific array data to fill form fields with. Like so for each needed value:
$lines = explode("\n", $html);
$data['vraagprijs'] = preg_replace("/[^0-9]/", "", $lines[5]);
However, the data i need may be in Line 10 today, but might very well be line 11 tomorrow. So I'd like to get the values into named arrays. A sample of the HTML on the URL is as follows:
<div class="item_list">
<span class="item first status">
<span class="itemName">Status</span>
<span class="itemValue">Sold</span>
</span>
<span class="item price">
<span class="itemName">Vraagprijs</span>
<span class="itemValue">389.000</span>
</span>
<span class="item condition">
<span class="itemName">Aanvaarding</span>
<span class="itemValue">In overleg</span>
</span>
...
</div>
This is my function model:
$tagName3 = 'div';
$attrName3 = 'class';
$attrValue3 = 'item_list';
$html = getShortTags($tagName3, $attrName3, $attrValue3, $url);
function getShortTags($tagName, $attrName, $attrValue, $url = "", $exclAttrValue = 'itemTitle') {
$dom = $this->getDom($url);
$html = '';
$domxpath = new \DOMXPath($dom);
$newDom = new \DOMDocument;
$newDom->formatOutput = true;
$filtered = $domxpath->query(" //" . $tagName . "[#" . $attrName . "='" . $attrValue . "']/descendant::text()[not(parent::span/#" . $attrName . "='" . $exclAttrValue . "')] ");
$i = 0;
while ($myItem = $filtered->item($i++)) {
$node = $newDom->importNode($myItem, true);
$newDom->appendChild($node);
}
$html = $newDom->saveHTML();
return $html;
}
What am I getting?
Status\nSold\nVraagprijs\n389.000\nIn overleg\n....
Desired output anything like:
$html = array("Status" => "Sold", "Vraagprijs" => "389.000", "Aanvaarding" => "In overleg", ...)
Is there a way to "loop" through the itemList and get each itemName and itemValue into an associative array?
If your happy with what the getShortTags() method does (or if it's used elsewhere and so difficult to tweak), then you can process the return value.
This code first uses explode() to split the output by line, uses array_map() and trim() to remove any spaces etc., then passes the result through array_filter() to remove blank lines. This will leave the data in pairs, so an easy way is to use array_chunk() to extract the pairs and then foreach() over the pairs with the first as the key and the second as the value...
$html = getShortTags($tagName3, $attrName3, $attrValue3, $url);
$lines = array_filter(array_map("trim", explode(PHP_EOL, $html)));
$pairs = array_chunk($lines, 2);
$output = [];
foreach ( $pairs as $pair ) {
$output[$pair[0]] = $pair[1];
}
print_r($output);
with the sample data gives..
Array
(
[Status] => Sold
[Vraagprijs] => 389.000
[Aanvaarding] => In overleg
)
To use this directly in the document and without making any assumptions (although if you don't have a name for several values, then not sure what you will end up with). This just looks specifically for the base element and then loops over the <span> elements. Each time within this it will look for the itemName and itemValue class attributes and get the value from these...
$output = [];
$filtered = $domxpath->query("//div[#class='item_list']/span");
foreach ( $filtered as $myItem ) {
$name= $domxpath->evaluate("string(descendant::span[#class='itemName'])", $myItem);
$value= $domxpath->evaluate("string(descendant::span[#class='itemValue'])", $myItem);
$output[$name] = $value;
}
print_r($output);
Testing with data scraping. The output I'm scraping, is a percent. So I basically slapped on a
echo "%<br>";
At the end of the actual number output which is
echo $ret_[66];
However there's an issue where the percent is actually appearing before the number as well, which is not desirable. This is the output:
%
-0.02%
Whereas what I'm trying to get is just -0.02%
Clearly I'm doing something wrong with the PHP. I'd really appreciate any feedback/solutions. Thank you!
Full code:
<?php
error_reporting(E_ALL^E_NOTICE^E_WARNING);
include_once "global.php";
$doc = new DOMDocument;
// We don't want to bother with white spaces
$doc->preserveWhiteSpace = false;
$doc->strictErrorChecking = false;
$doc->recover = true;
$doc->loadHTMLFile('http://www.moneycontrol.com/markets/global-indices/');
$xpath = new DOMXPath($doc);
$query = "//div[#class='MT10']";
$entries = $xpath->query($query);
foreach ($entries as $entry) {
$result = trim($entry->textContent);
$ret_ = explode(' ', $result);
//make sure every element in the array don't start or end with blank
foreach ($ret_ as $key => $val){
$ret_[$key] = trim($val);
}
//delete the empty element and the element is blank "\n" "\r" "\t"
//I modify this line
$ret_ = array_values(array_filter($ret_,deleteBlankInArray));
//echo the last element
echo $ret_[66];
echo "%<br>";
}
<?php
echo "%<br>";
?>
On a seperate following PHP code. Does the same thing.
I am new to DOM parsing, but I got most of this figured out. I'm just having trouble removing nbsp; from a div.
Here's my PHP:
function parseDOM($url) {
$dom = new DOMDocument;
#$dom->loadHTMLFile($url);
$xpath = new DOMXPath($dom);
$movies = array();
foreach ($xpath->query('//div[#class="mshow"]') as $movie) {
$item = array();
$links = $xpath->query('.//a', $movie);
$item['trailer'] = $links->item(0)->getAttribute('href');
$item['reviews'] = $links->item(1)->getAttribute('href');
$item['link'] = $links->item(2)->getAttribute('href');
$item['title'] = $links->item(2)->nodeValue;
$item['rating'] = trim($xpath->query('.//strong/following-sibling::text()',
$movie)->item(0)->nodeValue);
$i = 0;
foreach ($xpath->query('.//div[#class="rsd"]', $movie) as $date) {
$dates = $xpath->query('.//div[#class="rsd"]', $movie);
$times = $xpath->query('.//div[#class="rst"]', $movie);
$item['datetime'][] = $dates->item($i)->nodeValue . $times->item($i)->nodeValue;
$i += 1;
}
$movies[] = $item;
}
return $movies;
}
$url = 'http://www.tribute.ca/showtimes/theatres/may-cinema-6/mayc5/?datefilter=-1';
$movies = parseDOM($url);
foreach ($movies as $key => $value) {
echo $value['title'] . '<br>';
echo $value['link'] . '<br>';
echo $value['rating'] . '<br>';
foreach ($value['datetime'] as $datetime) {
echo $datetime . '<br>';
}
}
Here's what the HTML looks like:
<div class="rst" >6:45pm 9:30pm </div>
Is there something I can add to the xpath query to achieve this? I did try adding strip_tags to $times->item($i)->nodeValue, but it's still printing out like: Thu, May 01: 6:45pm   9:30pm  Â
Edit: str_replace("\xc2\xa0", '', $times->item($i)->nodeValue); seems to do the trick.
try this :
$times->item($i)->nodeValue = str_replace(" ","",$times->item($i)->nodeValue);
it should delete every
EDIT
your line :
$item['datetime'][] = $dates->item($i)->nodeValue . $times->item($i)->nodeValue;
become :
$item['datetime'][] = $dates->item($i)->nodeValue
. str_replace(" ","",$times->item($i)->nodeValue);
EDIT 2
if str_replace does not work, try with str_ireplaceas suggested in comment.
If it still doesn't work, you can also try with :
preg_replace("# #","",$times->item($i)->nodeValue);
EDIT 3
you may have an encoding problem. see uft8_encode
Or piggy solution :
str_replace("Â","",$times->item($i)->nodeValue);
Apolo
I want to retrieve an HTML element in a page.
<h2 id="resultCount" class="resultCount">
<span>
Showing 1 - 12 of 40,923 Results
</span>
</h2>
I have to get the total number of results for the test in my php.
For now, I get all that is between the h2 tags and I explode the first time with space.
Then I explode again with the comma to concatenate able to convert numbers results in European format. Once everything's done, I test my number results.
define("MAX_RESULT_ALL_PAGES", 1200);
$queryUrl = AMAZON_TOTAL_BOOKS_COUNT.$searchMonthUrlParam.$searchYearUrlParam.$searchTypeUrlParam.urlencode($keyword)."&page=".$pageNum;
$htmlResultCountPage = file_get_html($queryUrl);
$htmlResultCount = $htmlResultCountPage->find("h2[id=resultCount]");
$resultCountArray = explode(" ", $htmlResultCount[0]);
$explodeCount = explode(',', $resultCountArray[5]);
$europeFormatCount = '';
foreach ($explodeCount as $val) {
$europeFormatCount .= $val;
}
if ($europeFormatCount > MAX_RESULT_ALL_PAGES) {*/
$queryUrl = AMAZON_SEARCH_URL.$searchMonthUrlParam.$searchYearUrlParam.$searchTypeUrlParam.urlencode($keyword)."&page=".$pageNum;
}
At the moment the total number of results is not well recovered and the condition does not happen even when it should.
Someone would have a solution to this problem or any other way?
I would simply fetch the page as a string (not html) and use a regular expression to get the total number of results. The code would look something like this:
define('MAX_RESULT_ALL_PAGES', 1200);
$queryUrl = AMAZON_TOTAL_BOOKS_COUNT . $searchMonthUrlParam . $searchYearUrlParam . $searchTypeUrlParam . urlencode($keyword) . '&page=' . $pageNum;
$queryResult = file_get_contents($queryUrl);
if (preg_match('/of\s+([0-9,]+)\s+Results/', $queryResult, $matches)) {
$totalResults = (int) str_replace(',', '', $matches[1]);
} else {
throw new \RuntimeException('Total number of results not found');
}
if ($totalResults > MAX_RESULT_ALL_PAGES) {
$queryUrl = AMAZON_SEARCH_URL . $searchMonthUrlParam . $searchYearUrlParam . $searchTypeUrlParam . urlencode($keyword) . '&page=' . $pageNum;
// ...
}
A regex would do it:
...
preg_match("/of ([0-9,]+) Results/", $htmlResultCount[0], $matches);
$europeFormatCount = intval(str_replace(",", "", $matches[1]));
...
Please try this code.
define("MAX_RESULT_ALL_PAGES", 1200);
// new dom object
$dom = new DOMDocument();
// HTML string
$queryUrl = AMAZON_TOTAL_BOOKS_COUNT.$searchMonthUrlParam.$searchYearUrlParam.$searchTypeUrlParam.urlencode($keyword)."&page=".$pageNum;
$html_string = file_get_contents($queryUrl);
//load the html
$html = $dom->loadHTML($html_string);
//discard white space
$dom->preserveWhiteSpace = TRUE;
//Get all h2 tags
$nodes = $dom->getElementsByTagName('h2');
// Store total result count
$totalCount = 0;
// loop over the all h2 tags and print result
foreach ($nodes as $node) {
if ($node->hasAttributes()) {
foreach ($node->attributes as $attribute) {
if ($attribute->name === 'class' && $attribute->value == 'resultCount') {
$inner_html = str_replace(',', '', trim($node->nodeValue));
$inner_html_array = explode(' ', $inner_html);
// Print result to the terminal
$totalCount += $inner_html_array[5];
}
}
}
}
// If result count grater than 1200, do this
if ($totalCount > MAX_RESULT_ALL_PAGES) {
$queryUrl = AMAZON_SEARCH_URL.$searchMonthUrlParam.$searchYearUrlParam.$searchTypeUrlParam.urlencode($keyword)."&page=".$pageNum;
}
Give this a try:
$match =array();
preg_match('/(?<=of\s)(?:\d{1,3}+(?:,\d{3})*)(?=\sResults)/', $htmlResultCount, $match);
$europeFormatCount = str_replace(',','',$match[0]);
The RegEx reads the number between "of " and " Results", it matches numbers with ',' seperator.
The XML feed is located at: http://xml.betclick.com/odds_fr.xml
I need a php loop to echo the name of the match, the hour, and the bets options and the odds links.
The function will select and display ONLY the matchs of the day with streaming="1" and the bets type "Ftb_Mr3".
I'm new to xpath and simplexml.
Thanks in advance.
So far I have:
<?php
$xml_str = file_get_contents("http://xml.betclick.com/odds_fr.xml");
$xml = simplexml_load_string($xml_str);
// need xpath magic
$xml->xpath();
// display
?>
Xpath is pretty simple once you get the hang of it
you basically want to get every match tag with a certain attribute
//match[#streaming=1]
will work pefectly, it gets every match tag from underneath the parent tag with the attribute streaming equal to 1
And i just realised you also want matches with a bets type of "Ftb_Mr3"
//match[#streaming=1]/bets/bet[#code="Ftb_Mr3"]
This will return the bet node though, we want the match, which we know is the grandparent
//match[#streaming=1]/bets/bet[#code="Ftb_Mr3"]/../..
the two dots work like they do in file paths, and gets the match.
now to work this into your sample just change the final bit to
// need xpath magic
$nodes = $xml->xpath('//match[#streaming=1]/bets/bet[#code="Ftb_Mr3"]/../..');
foreach($nodes as $node) {
echo $node['name'].'<br/>';
}
to print all the match names.
I don't know how to work xpath really, but if you want to 'loop it', this should get you started:
<?php
$xml = simplexml_load_file("odds_fr.xml");
foreach ($xml->children() as $child)
{
foreach ($child->children() as $child2)
{
foreach ($child2->children() as $child3)
{
foreach($child3->attributes() as $a => $b)
{
echo $a,'="',$b,"\"</br>";
}
}
}
}
?>
That gets you to the 'match' tag which has the 'streaming' attribute. I don't really know what 'matches of the day' are, either, but...
It's basically right out of the w3c reference:
http://www.w3schools.com/PHP/php_ref_simplexml.asp
I am using this on a project. Scraping Beclic odds with:
<?php
$match_csv = fopen('matches.csv', 'w');
$bet_csv = fopen('bets.csv', 'w');
$xml = simplexml_load_file('http://xml.cdn.betclic.com/odds_en.xml');
$bookmaker = 'Betclick';
foreach ($xml as $sport) {
$sport_name = $sport->attributes()->name;
foreach ($sport as $event) {
$event_name = $event->attributes()->name;
foreach ($event as $match) {
$match_name = $match->attributes()->name;
$match_id = $match->attributes()->id;
$match_start_date_str = str_replace('T', ' ', $match->attributes()->start_date);
$match_start_date = strtotime($match_start_date_str);
if (!empty($match->attributes()->live_id)) {
$match_is_live = 1;
} else {
$match_is_live = 0;
}
if ($match->attributes()->streaming == 1) {
$match_is_running = 1;
} else {
$match_is_running = 0;
}
$match_row = $match_id . ',' . $bookmaker . ',' . $sport_name . ',' . $event_name . ',' . $match_name . ',' . $match_start_date . ',' . $match_is_live . ',' . $match_is_running;
fputcsv($match_csv, explode(',', $match_row));
foreach ($match as $bets) {
foreach ($bets as $bet) {
$bet_name = $bet->attributes()->name;
foreach ($bet as $choice) {
// team numbers are surrounded by %, we strip them
$choice_name = str_replace('%', '', $choice->attributes()->name);
// get the float value of odss
$odd = (float)$choice->attributes()->odd;
// concat the row to be put to csv file
$bet_row = $match_id . ',' . $bet_name . ',' . $choice_name . ',' . $odd;
fputcsv($bet_csv, explode(',', $bet_row));
}
}
}
}
}
}
fclose($match_csv);
fclose($bet_csv);
?>
Then loading the csv files into mysql. Running it once a minute, works great so far.