I have made a function that uses keywords like "Dell laptop x500" or something and it is trying to search for it. I did a hacky way of just adding it to the keywords search url, but it will give me different results compared to if i typed the text in the search box and pressed submit. Then it grabs the first results link back. Sometimes this works correctly and sometimes it does not.
function getAmazonLink($keywords){
$keywords = preg_replace("/[^a-z0-9_\s-]/", "%20", $keywords);
$link = "http://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Daps&field-keywords=$keywords";
//return $link;
$content = getContents($link);
$doc = new DOMDocument();
$doc->loadHTML($content);
$as = $doc->getElementsByTagName('a');
foreach ( $as as $a){
if($a->parentNode->nodeName == 'h3'){
if($a->parentNode->getAttribute('class') == 'newaps'){
if($a->parentNode->parentNode->getAttribute('id') == 'result_0'){
return $a->getAttribute('href');
}
}
}
}
return $link;
Amazon, like many other online stores, will tailor the search results based on your account's purchase/search history. Since your webapp isn't using a logged in Amazon account, it is getting results which aren't tailored to anyone's account history. In the comments you asked if there is a way to "work around this", but there's nothing to work around -- it's giving you valid results, just not ones which are tailored to a specific person's Amazon account. This is the expected result, not a bug.
Related
I'm using laravel to create a simple social network. Users can type # in the post area to get a list of their friends to mention them. Every mention in a link like this (using zurb/tribute from github)
<a type="mention" href="/user/Jordan" title="Jordan">Jordan</a>
Normal links other than mentions won't have type='mention'
Now when I get the post and insert it into the database I need to get a list of users mentioned in the post. I'm looking for links which have the type ='mention' and if there's any I want to get the title of everyone to insert into the notification system. What PHP code do I need to add in this if statement?
if(stristr(request('post'),' type="mention" ')){
}
Aside from using an AST (Abstract Syntax Tree), your best bet would be to either use DOM on PHP Side, e.g.:
$string = '<a type="mention" href="/user/Jordan" title="Jordan">Jordan</a>';
$doc = new \DOMDocument();
$doc->loadHTML($string);
$xpath = new \DOMXPath($doc);
foreach ($xpath->query("//a[#type='mention']") as $a) {
$href = $a->getAttribute('href');
$title = $a->getAttribute('title');
echo sprintf("Found mention of %s with href of %s\n", $title, $href);
}
However, I probably wouldn't be sending the A node back to the server. You should consider working out some way to make it a display-only feature implemented on the browser side, and simply send the "#jordan" string back to the server.
URL : http://www.sayuri.co.jp/used-cars
Example : http://www.sayuri.co.jp/used-cars/B37753-Toyota-Wish-japanese-used-cars
Hey guys , need some help with one of my personal projects , I've already wrote the code to fetch data from each single car url (example) and post on my site
Now i need to go through the main url : sayuri.co.jp/used-cars , and :
1) Make an array / list / nodes of all the urls for all the single cars in it , then run my internal code for each one to fetch data , then move on to the next one
I already have the code to save each url into a log file when completed (don't think it will be necessary if it goes link by link without starting from the top but will ensure no repetition.
2) When all links are done for the page , it should move to the next page and do the same thing until the end ( there are 5-6 pages max )
I've been stuck on this part since last night and would really appreciate any help . Thanks
My code to get data from the main url :
$content = file_get_contents('http://www.sayuri.co.jp/used-cars/');
// echo $content;
and
$dom = new DOMDocument;
$dom->loadHTML($content);
//echo $dom;
I'm guessing you already know this since you say you've gotten data from the car entries themselves, but a good point to start is by dissecting the page's DOM and seeing if there are any elements you can use to jump around quickly. Most browsers have page inspection tools to help with this.
In this case, <div id="content"> serves nicely. You'll note it contains a collection of tables with the required links and a <div> that contains the text telling us how many pages there are.
Disclaimer, but it's been years since I've done PHP and I have not tested this, so it is probably neither correct or optimal, but it should get you started. You'll need to tie the functions together (what's the fun in me doing it?) to achieve what you want, but these should grab the data required.
You'll be working with the DOM on each page, so a convenience to grab the DOMDocument:
function get_page_document($index) {
$content = file_get_contents("http://www.sayuri.co.jp/used-cars/page:{$index}");
$document = new DOMDocument;
$document->loadHTML($content);
return $document;
}
You need to know how many pages there are in total in order to iterate over them, so grab it:
function get_page_count($document) {
$content = $document->getElementById('content');
$count_div = $content->childNodes->item($content->childNodes->length - 4);
$count_text = $count_div->firstChild->textContent;
if (preg_match('/Page \d+ of (\d+)/', $count_text, $matches) === 1) {
return $matches[1];
}
return -1;
}
It's a bit ugly, but the links are available inside each <table> in the contents container. Rip 'em out and push them in an array. If you use the link itself as the key, there is no concern for duplicates as they'll just rewrite over the same key-value.
function get_page_links($document) {
$content = $document->getElementById('content');
$tables = $content->getElementsByTagName('table');
$links = array();
foreach ($tables as $table) {
if ($table->getAttribute('class') === 'itemlist-table') {
// table > tbody > tr > td > a
$link = $table->firstChild->firstChild->firstChild->firstChild->getAttribute('href');
// No duplicates because they just overwrite the same entry.
$links[$link] = "http://www.sayuri.co.jp{$link}";
}
}
return $links;
}
Perhaps also obvious, but these will break if this site changes their formatting. You'd be better off asking if they have a REST API or some such available for long term use, though I'm guessing you don't care as much if it's just a personal project for tinkering.
Hope it helps prod you in the right direction.
What I want to accomplish might be a little hardcore, but I want to know if it's possible:
The question:
My question is the same as PHP-Retrieve content from page, but I want to use it on multiple pages.
The situation:
I'm using a website about TV shows. All the TV shows have the same URL and then the name of the show:
http://bierdopje.com/shows/NAME_OF_SHOW
On every show page, there's a line which tells you if the show is cancelled or still running. I want to retrieve that line to make an overview of the cancelled shows (the website only supports an overview of running shows, so I want to make an extra functionality).
The real question:
How can I tell DOM to retrieve all the shows and check for the status of the show?
(http://bierdopje.com/shows/*).
The Note:
I understand that this process may take a while because it is reading the whole website (or is it too much data?).
use this code to fetch only the links from the single website.
include_once('simple_html_dom.php');
$html = file_get_html('http://www.couponrani.com/');
// Find all links
foreach($html->find('a') as $element)
echo $element->href . '<br>';
I use phpquery to fetch data from a web page, like jQuery in Dom.
For example, to get the list of all shows, you can do this :
<?php
require_once 'phpQuery/phpQuery/phpQuery.php';
$doc = phpQuery::newDocumentHTML(
file_get_contents('http://www.bierdopje.com/shows')
);
foreach (pq('.listing a') as $key => $a) {
$url = pq($a)->attr('href'); // will give "/shows/07-ghost"
$show = pq($a)->text(); // will give "07 Ghost"
}
Now you can process all shows individualy, make a new phpQuery::newDocumentHTML for each show and with an selector extract the information you need.
Get the status of a show
$html = file_get_contents('http://www.bierdopje.com/shows/alcatraz');
$doc = phpQuery::newDocumentHTML($html);
$status = pq('.content>span:nth-child(6)')->text();
As recently as two days ago, the following code worked to get the search query from google:
$refer = parse_url($_SERVER['HTTP_REFERER']);
$host = parse_url($_SERVER['HTTP_REFERER'], PHP_URL_HOST);
$query = parse_url($_SERVER['HTTP_REFERER'], PHP_URL_QUERY);
if(strstr($host,'www.google.com'))
{
//do google stuff
$qstart = strpos($query, 'q=') +2;
$qend = strpos($query, '&', $qstart);
$qlength = $qend - $qstart;
$querystring = substr($query, $qstart, $qlength);
$querystring = str_replace('q=','',$querystring);
$keywords = explode('%20',$querystring);
$keywords = implode(' ', $keywords);
return $keywords;
}
However, now it does not. I tested it by using echo($query) and it appears that the way google processes referrer query requests has changed. Previously $query included
"q=term1%20term2%20term3%20...
Now, however, I am getting the following output when $query is echo'ed:
sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CCsQFjAB&url=http%3A%2F%2Fexample.com%2F&ei=vDA-UNnxHuOjyAHlloGYCA&usg=AFQjCNEvzNXHULR0OvoPMPSWxIlB9-fmpg&sig2=iPinsBaCFuhCLGFf0JHAsQ
Is there a way to get around this?
Sorry to say, but it's global Google politics change.
See web link
http://googlewebmastercentral.blogspot.ru/2012/03/upcoming-changes-in-googles-http.html
This means if user sign in Google account.
You can try it yourself: if your Google search url starts with https:// this means Google will hide some scratch parameters for the sake of privacy.
I too ran into the same issue this week. I'm not sure if this is still relevant to you, but what I found was that Google initiated SSL (Secure Sockets Layer) search for users who were signed in about a year ago, and it looks like SSL search may now be applied to all Google search queries. When I tested this, I was not signed in to Google and was using Firefox and still got the encrypted referring query.
This article has some helpful background and some ideas for working without specific search term data: http://searchenginewatch.com/article/2227114/5-Tips-for-Handling-Not-Provided-Data
// take the referer
$thereferer = strtolower($_SERVER['HTTP_REFERER']);
// see if it comes from google
if (strpos($thereferer,"google")) {
// delete all before q=
$a = substr($thereferer, strpos($thereferer,"q="));
// delete q=
$a = substr($a,2);
// delete all FROM the next & onwards
if (strpos($a,"&")) {
$a = substr($a, 0,strpos($a,"&"));
}
// we have the results.
$mygooglekeyword = urldecode($a);
}
Google initiated SSL for all searches and the information is only available via Google Analytics.
However, for paid campaigns search engines like Google, Bing and Yahoo use query string parameters such as utm_parameters and you can access the search query from the parameter utm_term.
This may be a stupid question, but is it possible to capture what a user typed into a Google search box, so that this can then be used to generate a dynamic page on the landing page on my Web site?
For example, let's say someone searches Google for "hot dog", and my site comes up as one of the search result links. If the user clicks the link that directs them to my Web site, is it possible for me to somehow know or capture the "hot dog" text from the Google search box, so that I can call a script that searches my local database for content related to hot dogs, and then display that? It seems totally impossible to me, but I don't really know. Thanks.
I'd do it like this
$referringPage = parse_url( $_SERVER['HTTP_REFERER'] );
if ( stristr( $referringPage['host'], 'google.' ) )
{
parse_str( $referringPage['query'], $queryVars );
echo $queryVars['q']; // This is the search term used
}
This is an old question and the answer has changed since the original question was asked and answered. As of October 2011 Google is encrypting this referral information for anyone who is logged into a Google account: http://googleblog.blogspot.com/2011/10/making-search-more-secure.html
For users not logged into Google, the search keywords are still found in the referral URL and the answers above still apply. However, for authenticated Google users, there is no way to for a website to see their search keywords.
However, by creating dedicated landing pages it might still be possible to make an intelligent guess. (Visitors to the "Dignified charcoal sketches of Jabba the Hutt" page are probably...well, insane.)
Yes, it is possible. See HTTP header Referer. The Referer header will contain URL of Google search result page.
When user clicks a link on a Google search result page, the browser will make a request to your site with this kind of HTTP header:
Referer: http://www.google.fi/search?hl=en&q=http+header+referer&btnG=Google-search&meta=&aq=f&oq=
Just parse URL from request header, the search term used by user will be in q -parameter. Search term used in above example is "http header referer".
Same kind of approach usually works also for other search engines, they just have different kind of URL in Referer header.
This answer shows how to implement this in PHP.
Referer header is only available with HTTP 1.1, but that covers just about any somewhat modern browser. Browser may also forge Referer header or the header might be missing altogether, so do not make too serious desicions based on Referer header.
This is an old question but I found out that google no more gives out the query term because it by default redirects every user to https which will not give you the "q"parameter. Unless someone manually enters the google url with http (http://google.com) and then searches, there is no way as of now to get the "q" parameter.
Yes, it comes in the url:
http://www.google.com/search?hl=es&q=hot+dog&lr=&aq=f&oq=
here is an example:
Google sends many visitors to your site, if you want to get the keywords
they used to come to your site, maybe to impress them by displaying it
back on the page, or just to store the keyword in a database, here's the
PHP code I use :
// take the referer
$thereferer = strtolower($_SERVER['HTTP_REFERER']);
// see if it comes from google
if (strpos($thereferer,"google")) {
// delete all before q=
$a = substr($thereferer, strpos($thereferer,"q="));
// delete q=
$a = substr($a,2);
// delete all FROM the next & onwards
if (strpos($a,"&")) {
$a = substr($a, 0,strpos($a,"&"));
}
// we have the results.
$mygooglekeyword = urldecode($a);
}
and we can use <?= $mygooglekeywords ?> when we want to output the
keywords.
You can grab the referring URL and grab the search term from the query string. The search will be in the query as "q=searchTerm" where searchTerm is the text you want.
Same thing, but with some error handling
<?php
if (#$_SERVER['HTTP_REFERER']) {
$referringPage = parse_url($_SERVER['HTTP_REFERER']);
if (stristr($referringPage['host'], 'google.')) {
parse_str( $referringPage['query'], $queryVars );
$google = $queryVars['q'];
$google = str_replace("+"," ",$google); }
else { $google = false; }}
else { $google = false; }
if ($google) { echo "You searched for ".$google." at Google then came here!"; }
else { echo "You didn't come here from Google"; }
?>
Sorry, a little more
Adds support for Bing, Yahoo and Altavista
<?php
if (#$_SERVER['HTTP_REFERER']) {
$referringPage = parse_url($_SERVER['HTTP_REFERER']);
if (stristr($referringPage['host'], 'google.')
|| stristr($referringPage['host'], 'bing.')
|| stristr($referringPage['host'], 'yahoo.')) {
parse_str( $referringPage['query'], $queryVars );
if (stristr($referringPage['host'], 'google.')
|| stristr($referringPage['host'], 'bing.')) { $search = $queryVars['q']; }
else if (stristr($referringPage['host'], 'yahoo.')) { $search = $queryVars['p']; }
else { $search = false; }
if ($search) { $search = str_replace("+"," ",$search); }}
else { $search = false; }}
else { $search = false; }
if ($search) { echo "You're in the right place for ".$search; }
?>