I'm new to Regular Expressions and things like that. I have only few knowledge and I think my current problem is about them.
I have a webpage, that contains text. I want to get links from the webpage that are only in SPANs that have class="img".
I go through those steps.
grab all the SPANs tagged with the "img" class (this is the hard step that I'm looking for)
move those SPANs to a new variable
Parse the variable to get an array with the links (Each SPAN has only 1 link, so this will be easy)
I'm using PHP, but any other language doesn't matter, I'm looking how to deal with the first step. Any one have a suggestion?
Thanks :D
Use PHPs DOMDocument-class in combination with the DOMXPath-class to navigate to the elements you need, like this:
<?php
$dom = new DOMDocument();
$dom->loadHTML(file_get_contents('http://foo.bar'));
$xpath = new DOMXPath($dom);
$elements = $xpath->query("/html/body//span[#class='img']//a");
foreach ($elements as $a)
{
echo $a->getAttribute('href'), "\n";
}
You can learn more about the XPath Language on the W3C page.
A pattern like <span.* class="img".*>([^<]*)</span> should work fine., assuming your code looks something like
<span class="img">http://www.img.com/img.jpg</span>
<span alt="yada" class="img">animage.png</span>
<span alt="yada" class="img" title="still works">link.txt</span>
<span>not an img class</span>
<?php
$pattern = '#<span.* class="img".*>([^<]*)</span>#i';
//$subject = html code above
preg_match_all($pattern, $subject, $matches);
print_r($matches);
?>
I'm using PHP, but any other language
doesn't matter, I'm looking how to
deal with the first step. Any one have
a suggestion?
We-e-ell...
import urllib
from BeautifulSoup import BeautifulSoup, SoupStrainer
html = urllib.urlopen(url).read()
sieve = SoupStrainer(name='span', attrs={'class': 'img'})
tag_soup = BeautifulSoup(html, parseOnlyThese=sieve)
for link in tag_soup('a'):
print link['href']
(that's python, using BeautifulSoup - should work on most douments, well-formed or no).
Related
I have a line of sourcecode looking like this
<img alt="this field is variable" title="this one too" itemprop="photo" border="0" style="width:608px;" src="imgurl.jpg">
There's lots of other images on the site, so i can't just preg_match all images, i need the specific one, i had a lot of trouble doing a specific preg_match, because content of the "alt"-tag and "title"-tag is variable. Anyone knows how to do so? Thanks in advance.
Itemprop="photo" is the thing unique for this picture.
This regex should work:
preg_match('/<img[^>]*itemprop="photo"[^>]*src="([^"]+)">/',$source,$matches);
An explanation of the regex (from regex101):
The result will be in the array $matches.
Using regex to parse HTML is not a good thing. Why not use DOMDocument to search for your elements? PHP has these objects for parsing through an HTML document and examining elements much easier than using regex to try to find them. You would also then be able to manipulate the HTML much easier depending on what it is that you are trying to accomplish.
$dom = new DOMDocument();
$dom->loadHTML(<your html string>);
$imgs = $dom->getElementsByTagName('img');
$photos = [];
foreach($imgs as $img) {
if($img->attributes->getNamedItem('itemprop') && $img->attributes->getNamedItem('itemprop')->nodeValue = 'photo') {
$photos[] = $img->attributes->getNamedItem('src')->nodeValue;
}
}
This code will get you an array with the src attribute of the imgs that have your property and you are not dependent on how the elements are created or anything in the actual text of the html.
I am Expanding/Condensing a Blog post using substrings, where the second substring is within a div tag that activates when a button is pressed (hence concatenating both substrings)
The code looks like as below:
<?php echo substr($f2, 0, 50);?>
<div id="<?php echo $f4; ?>" class = "hidden">
<?php echo substr($f2, 0, 5000);?></div>
My problem however is if the blog post contains html tags (e.g. <\li>, <\p>) and the initial substring ends before the termination of that set of tags, then obviously it causes major formatting problems.
Is there a way around this using my current method, or am I going to need to use something like an XML stylesheet (in which case please guide me through it)
EDIT:
I have semi-completed my request using DOMDocument.
$second = substr($f2, 50, 5000);
$dom= new DOMDocument();
$dom->loadHTML($second);
$xpath = new DOMXPath($dom);
$body = $xpath->query('/html/body');
$secondoutput = ($dom->saveXml($body->item(0)));
$first = substr($f2, 0, 50);
$dom= new DOMDocument();
$dom->loadHTML($first);
$xpath = new DOMXPath($dom);
$body = $xpath->query('/html/body');
$firstoutput = ($dom->saveXml($body->item(0)));
This works except, when the second subtring is called it no longer has the previous formatting as it has been purified.
Is there any way to reattch the previous HTML tag when the second substring is called?
There are diffrent solutions to this problem, but substr is not particularly suitable (as you mentioned).
You could use Regular Expressions, or a HTML-Parser.
Go ahead and copy solutions from this question.
You may want to use Tidy to fix the truncated HTML.
You may want to parse whole HTML code with DOMDocument or SimpleHTMLDOM and then remove last elements until the post is short enough.
After using curl i've got from an external page i've got all source code with something like this (the part i'm interested)
(page...)<td valign='top' class='rdBot' align='center'><img src="/images/buy_tickets.gif" border="0" alt="T"></td> (page...)
So i'm using preg_match_all, i want to get only "buy_tickets.gif"
$pattern_before = "<td valign='top' class='rdBot' align='center'>";
$pattern_after = "</td>";
$pattern = '#'.$pattern_before.'(.*?)'.$pattern_after.'#si';
preg_match_all($pattern, $buffer, $matches, PREG_SET_ORDER);
Everything fine up to now... but the problem it's becase sometimes that external pages changes and the image i'm looking for it's inside a link
(page...)<td valign='top' class='rdBot' align='center'><img src="/images/buy_tickets.gif" border="0" alt="T"></td> (page...)
and i dunno how to get always my code to work (not just when the image gets no link)
hope u understand
thanks in advance
Don't use regex to parse HTML, Use PHP's DOM Extension. Try this:
$doc = new DOMDocument;
#$doc->loadHTMLFile( 'http://ventas.entradasmonumental.com/eventperformances.asp?evt=18' ); // Using the # operator to hide parse errors
$xpath = new DOMXPath( $doc );
$img = $xpath->query( '//td[#class="BrdBot"][#align="center"][1]//img[1]')->item( 0 ); // Xpath->query returns a 'DOMNodeList', get the first item which is a 'DOMElement' (or null)
$imgSrc = $img->getAttribute( 'src' );
$imgSrcInfo = pathInfo( $imgSrc );
$imgFilename = $imgSrcInfo['basename']; // All you need
You're going to get lots of advice not to use regex for pulling stuff out of HTML code.
There are times when it's appropriate to use regex for this kind of thing, and I don't always agree with the somewhat rigid advice given on the subject here (and elsewhere). However in this case, I would say that regex is not the appropriate solution for you.
The problem with using regex for searching for things in HTML code is exactly the problem you've encountered -- HTML code can vary wildly, making any regex virtually impossible to get right.
It is just about possible to write a regex for your situation, but it will be an insanely complex regex, and very brittle -- ie prone to failing if the HTML code is even slightly outside the parameters you expect.
Contrast this with the recommended solution, which is to use a DOM parser. Load the HTML code into a DOM parser, and you will immediately have an object structure which you can query for individual elements and attributes.
The details you've given make it almost a no-brainer to go with this rather than a regex.
PHP has a built-in DOM parser, which you can call as follows:
$mydom = new DOMDocument;
$mydom->loadHTMLFile("http://....");
You can then use XPath to search the DOM for your specific element or attribute that you want:
$myxpath = new DOMXPath($mydom);
$myattr = $xpath->query("//td[#class="rdbot"]//img[0]#src");
Hope that helps.
function GetFilename($file) {
$filename = substr($file, strrpos($file,'/')+1,strlen($file)-strrpos($file,'/'));
return $filename;
}
echo GetFilename('/images/buy_tickets.gif');
This will output buy_tickets.gif
Do you only need images inside of the "td" tags?
$regex='/<img src="\/images\/([^"]*)"[^>]*>/im';
edit:
to grab the specific image this should work:
$regex='/<td valign=\'top\' class=\'rdBot\' align=\'center\'>.*src="\/images\/([^"]*)".*<\/td>/
Parsing HTML with Regex is not recommended, as has been mentioned by several posters.
However, if the path of your images always follows the pattern src="/images/name.gif", you can easily extract it in Regex:
$pattern = <<<EOD
#src\s*=\s*['"]/images/(.*?)["']#
EOD;
If you are sure that the images always follow the path "/images/name.ext" and that you don't care where the image link is located in the page, this will do the job. If you have more detailed requirements (such matching only within a specific class), forget Regex, it's not the right tool for the job.
I just read in your comments that you need to match within a specific tag. Use a parser, it will save you untold headaches.
If you still want to go through regex, try this:
\(?<=<td .*?class\s*=\s*['"]rdBot['"][^<>]*?>.*?)(?<!</td>.*)<img [^<>]*src\s*=\s*["']/images/(.*?)["']\i
This should work. It does work in C#, I am not totally sure about php's brand of regex.
I'm using PHP's "simplexml_load_file" to get some data from Flickr.
My goal is to get the photo url.
I'm able to get the following value (assigned to PHP variable):
<p>codewrecker posted a photo:</p>
<p><img src="http://farm3.static.flickr.com/2298/2302759205_4fb109f367_m.jpg" width="180" height="240" alt="Santa Monica Pier" /></p>
How can I extract just this part of it?
http://farm3.static.flickr.com/2298/2302759205_4fb109f367_m.jpg
Just in case it helps, here's the code I'm working with:
<?php
$xml = simplexml_load_file("http://api.flickr.com/services/feeds/photos_public.gne?id=19725893#N00&lang=en-us&format=xml&tags=carousel");
foreach($xml->entry as $child) {
$flickr_content = $child->content; // gets html including img url
// how can I get the img url from "$flickr_content"???
}
?>
You can probably get away with using a regular expression for this, assuming that the way the HTML is formed is pretty much going to stay the same, e.g.:
if (preg_match('/<img src="([^"]+)"/i', $string, $matches)) {
$imageUrl = $matches[1];
}
This is fairly un-robust, and if the HTML is going to change (e.g. the order of parameters in the <img> tag, risk of malformed HTML etc.), you would be better off using an HTML parser.
It's not solving your problem(and probably total overkill), but worth mentioning because I've used the library on 2 projects and it's well written.
phpFlickr - http://phpflickr.com/
Easy way: Combination of substr and strpos to extract first the tag and then the src='...' value, and finally the target string.
Slightly more difficult way (BUT MUCH MORE ROBUST): Use an XML parsing library such as simpleXML
I hope this is helpful. I enjoy using xpath to cut through the XML I get back from SimpleXML:
<?php
$xml = new SimpleXMLElement("http://api.flickr.com/services/feeds/photos_public.gne?id=19725893#N00&lang=en-us&format=xml&tags=carousel", NULL, True);
$images = $xml->xpath('//img'); //use xpath on the XML to find the img tags
foreach($images as $image){
echo $image['src'] ; //here is the image URL
}
?>
I am trying to create a simple alert app for some friends.
Basically i want to be able to extract data "price" and "stock availability" from a webpage like the folowing two:
http://www.sparkfun.com/commerce/product_info.php?products_id=5
http://www.sparkfun.com/commerce/product_info.php?products_id=9279
I have made the alert via e-mail and sms part but now i want to be able to get the quantity and price out of the webpages (those 2 or any other ones) so that i can compare the price and quantity available and alert us to make an order if a product is between some thresholds.
I have tried some regex (found on some tutorials, but i an way too n00b for this) but haven't managed to get this working, any good tips or examples?
$content = file_get_contents('http://www.sparkfun.com/commerce/product_info.php?products_id=9279');
preg_match('#<tr><th>(.*)</th> <td><b>price</b></td></tr>#', $content, $match);
$price = $match[1];
preg_match('#<input type="hidden" name="quantity_on_hand" value="(.*?)">#', $content, $match);
$in_stock = $match[1];
echo "Price: $price - Availability: $in_stock\n";
It's called screen scraping, in case you need to google for it.
I would suggest that you use a dom parser and xpath expressions instead. Feed the HTML through HtmlTidy first, to ensure that it's valid markup.
For example:
$html = file_get_contents("http://www.example.com");
$html = tidy_repair_string($html);
$doc = new DomDocument();
$doc->loadHtml($html);
$xpath = new DomXPath($doc);
// Now query the document:
foreach ($xpath->query('//table[#class="pricing"]/th') as $node) {
echo $node, "\n";
}
What ever you do: Don't use regular expressions to parse HTML or bad things will happen. Use a parser instead.
1st, asking this question goes too into details. 2nd, extracting data from a website might not be legitimate. However, I have hints:
Use Firebug or Chrome/Safari Inspector to explore the HTML content and pattern of interesting information
Test your RegEx to see if the match. You may need do it many times (multi-pass parsing/extraction)
Write a client via cURL or even much simpler, use file_get_contents (NOTE that some hosting disable loading URLs with file_get_contents)
For me, I'd better use Tidy to convert to valid XHTML and then use XPath to extract data, instead of RegEx. Why? Because XHTML is not regular and XPath is very flexible. You can learn XSLT to transform.
Good luck!
You are probably best off loading the HTML code into a DOM parser like this one and searching for the "pricing" table. However, any kind of scraping you do can break whenever they change their page layout, and is probably illegal without their consent.
The best way, though, would be to talk to the people who run the site, and see whether they have alternative, more reliable forms of data delivery (Web services, RSS, or database exports come to mind).
The simplest method to extract data from Website. I've analysed that my all data is covered within <h3> tag only, so I've prepared this one.
<?php
include(‘simple_html_dom.php’);
// Create DOM from URL, paste your destined web url in $page
$page = ‘http://facebook4free.com/category/facebookstatus/amazing-facebook-status/’;
$html = new simple_html_dom();
//Within $html your webpage will be loaded for further operation
$html->load_file($page);
// Find all links
$links = array();
//Within find() function, I have written h3 so it will simply fetch the content from <h3> tag only. Change as per your requirement.
foreach($html->find(‘h3′) as $element)
{
$links[] = $element;
}
reset($links);
//$out will be having each of HTML element content you searching for, within that web page
foreach ($links as $out)
{
echo $out;
}
?>