I want to scrape some data from webpage that is Cross Reference and Model Information
I have a script which scraped all other data form page.
But as required data is coming from Ajax call in Div by id.
<div class="inner" style="overflow-x: hidden; overflow-y: scroll; height:300px;" id="xmodellist"><table>
<tbody><tr>
<th>Manufacturer</th>
<th>Model Number</th>
<th>Description</th>
</tr>
<tr>
<td>Amana</td>
<td>SXD26VW</td>
<td>REF - SXS/I&W</td>
</tr></tbody></table>
</div>
Ajax Code of webpage from view source
$.ajax({
url: "/partmodellist.aspx?partid=" + partid + "&os=" + os.toString(),
success: function (data) {
if (data) {
$('#xmodel1').replaceWith(data);
$('#xmodellist').scroll(function () {
if (($('tr#trxrefloading').length > 0) && $('#xmodellist').scrollTop() + $('#xmodellist').innerHeight() >= $('#xmodellist')[0].scrollHeight) {
os = os + 1;
$.ajax({
url: "/partmodellist.aspx?partid=" + partid + "&os=" + os.toString(),
success: function (html) {
if (html) {
if (html.trim().length > 0)
{
$("tr#trxrefloading").before(html);
}
else
{
$('tr#trxrefloading').css("display", "none");
$('#xmodellist').unbind();
}
} else {
$('tr#trxrefloading').css("display", "none");
$('#xmodellist').unbind();
}
}
});
}
});
} else {
$('xmodel1').css("display", "none");
}
}
});
I want to scrape all information in that div.
I have attached my existing code file. or this link
Please provide suggestion that how can I achieve this.
You can directly use CURL to simulation request to get data from partmodellist.aspx rather than scrape it or simulate browser behavior by selenium lib.
What is Selenium?
I was also trying to scrape ajax in php and found this:
http://256cats.com/scraping-asp-websites-php-dopostback-ajax-emulation/
I tried to implement it but I'm not a developer so my skills are limited and I am actually trying to do something much simpler than the example shown in the article. By that I mean: I couldn't do it. I also tried contacting the developer with no success.
But it might interest you.
Related
I have made a table which is dynamicly created using PHP.
It basicly looks like this.
<table class="persist-area table table-bordered ">
<thead>
<tr class="persist-header">
<th>1. th</th>
<th>2. th</th>
<th>3. th</th>
<th>4. th</th>
</tr>
</thead>
<tbody class="maintable">
<span class="genTable" id="<?php echo $partnerId ?>">
</thead>
<tbody class="maintable">
<span class="genTable" id="<?php echo $partnerId ?>">';
<?php echo $partner->genTable();?>
</span>
</tbody>
</table>
</div>
</div>
I have this directly in my partner.php file.
Now, I want to create an AJAX function, that loads this table when you visit partner.php and so that I am able to call the function, everytime the user makes changes to the table.
So I created this javascript function:
function genTable(){
$.ajax({ type: "POST",
url: "resources/dialogs/genTable.php",
data: {'partnerId': '1'},
success: function(msg){
$("div#gentable").html(msg);
}});
}
So this makes a POST request to a new file I created, which returns the exact same table as above.
But now my other javascripts does not work on the table anymore.
For instance, I have another script running, to make the table headers persistant on scroll :
function UpdateTableHeaders() {
$(".persist-area").each(function() {
var el = $(this),
offset = el.offset(),
scrollTop = $(window).scrollTop(),
floatingHeader = $(".floatingHeader", this)
if ((scrollTop > offset.top) && (scrollTop < offset.top + el.height())) {
floatingHeader.css({
"visibility": "visible"
});
} else {
floatingHeader.css({
"visibility": "hidden"
});
};
});
}
// DOM Ready
$(function() {
var clonedHeaderRow;
$(".persist-area").each(function() {
clonedHeaderRow = $(".persist-header", this);
clonedHeaderRow
.before(clonedHeaderRow.clone())
.css("width", clonedHeaderRow.width())
.addClass("floatingHeader");
});
$(window)
.scroll(UpdateTableHeaders)
.trigger("scroll");
});
The code above is just an example. All other javascripts that interact with the table does not work either - is there a way I can make this work?
I'm thinking it has something to do with how the stuff gets loaded? Should I use livequery or how to make my other scripts work on the AJAX generated content?
As said, all the scripts works fine if I load the table directly from partner.php file, but when trying to load it with AJAX it breaks.
Any help will be very appreciated.
This is actually simpler than you think.
All you need to do is put your listener in a function and re-call that function every time your ajax load is complete.
I want the div to change color based on the value that is retrieved from the database.
The current code doesn't give any color and uses the set css color.
Main div css
.status{ background: #000; }
status.php
if ($user_data['status'] === "1") { echo "online"; }
else { echo "offline"; }
The ajax function
$(document).ready(function() {
setInterval(function(){
$.ajax({
url: 'status.php',
type: 'GET',
success: function(data) {
if (data === "online") {
$('.status').css('background', '#40A547');
} else {
$('.status').css('background', '#7f8c8d');
}
}
});
}, 2000);
});
The main div
<div class="status">something</div>
Please help me out. All help will be much appreciated!
The problem can be identified use this code :
$(document).ready(function() {
setInterval(function(){
$.ajax({
url:'status.php',
datatype:"application/json",
type:'GET',
success:function(data){
if (data === "online") {
$('.status').css('background', '#40A547');
} else {
$('.status').css('background', '#7f8c8d');
}
},
error:function(){
//if pointer is comming it means there is some error in the back end code
}
});
}, 2000);
});
Now check your in the firebug it will show you the error details, if you don't have firebug add it and then check.
also put some content in the div so that it will be visible. If there are no content div doesn't appear.
Hope this will help!
You don't have anything in your div, so it's basically 0 height and width so you don't see anything (which is what I assume you mean by "current code doesn't give any color").
If you specify a height and width, or add something inside your div (even a , you should see the background color.
That means you are not getting 'online' back from your ajax. you need to edit your PHP code that you are AJAXing to purely say 'online', no divs or markup.
I'm working with AJAX on a website and I'm currently making some pages to load on a certain div: "pageContent". Now I have another content I want to be opened on another div: "reproductor". I want to open 'page' in 'pageContent' div and 'play' in 'reproductor' div. I don't know how to modify my script.js and load_page.php files in order to make it work. Here's what I got:
HTML:
<script type="text/javascript" src="js/script.js"></script>
PAGE
PLAY
<div id ="pageContent"></div>
<div id="reproductor"></div>
script.js:
var default_content="";
$(document).ready(function(){
checkURL();
$('ul li a').click(function (e){
checkURL(this.hash);
});
default_content = $('#pageContent').html();
setInterval("checkURL()",250);
});
var lasturl="";
function checkURL(hash)
{
if(!hash) hash=window.location.hash;
if(hash != lasturl)
{
lasturl=hash;
if(hash=="")
$('#pageContent').html(default_content);
else
loadPage(hash);
}
}
function loadPage(url)
{
url=url.replace('#page','');
$('#loading').css('visibility','visible');
$.ajax({
type: "POST",
url: "load_page.php",
data: 'page='+url,
dataType: "html",
success: function(msg){
if(parseInt(msg)!=0)
{
$('#pageContent').html(msg);
$('#loading').css('visibility','hidden');
}
}
});
}
load_page.php:
<?php
if(!$_POST['page']) die("0");
$page = (int)$_POST['page'];
if(file_exists('pages/page_'.$page.'.html'))
echo file_get_contents('pages/page_'.$page.'.html');
else
echo 'There is no such page!';
?>
I forgot to mention: I have my 'pages' content in a folder named 'pages' and my 'play' content in another named 'plays'.
Thanks for your help!
The easiest way to load content from a resource that serves HTML into an element is to use load:
$('#reproductor').load('public_html/plays/play_1.html', function(){
//stuff to do after load goes here
});
You could also apply this technique to the other div you are trying to load content into.
If I understand, your have two groups of links (for pages and a play list) each one to be loaded in a different container. Here is something you can try: mainly I eliminated the global variables and put the current hash inside each containter's data, and separated the management of the two groups of links.
In this code I supposed you have a separate load_play.php file. If not, then you can use the same page for both kind of links, but you'll have to merge loadPlay with loadPage, change loadPage(newHash) to loadPage(newHash, linkType) and change the ajax parameter from 'page='+newHash to 'number='+newHash+'&type='+linkType, and do the corresponding changes server side in your PHP page. I would recommend you to create two separate PHP files in order to manage the two types of content.
I remember you where doing something with the hash of the current page's url, you can still set it in the ajax's success, inside the loadPage function.
Here is a working sfiddle example with some console calls (open browser's console) but without the ajax call.
UPDATE:
I updated the code, so your can manage the dynamically added links (new content loaded via AJAX) and fixed the management of urls with hashes, which was broken because of the new code.
<div id="#page">
PAGE 1
PAGE 2
PLAY 1
PLAY 2
PLAY 3
<div id ="pageContent"></div>
<div id="reproductor"></div>
</div>
And this is the javascript:
$(document).ready(function(){
$('#pageContent').data('currentPage', '');
$('#reproductor').data('currentPlay', '');
//This will allow it to work even on dynamically created links
$('#page').on('click', '.pageLink', function (e){
loadPage(this.hash);
});
$('#page').on('click', '.playLink', function (e){
loadPlay(this.hash);
});
//And this is for managing the urls with hashes (for markers)
var urlLocation = location.hash;
if(urlLocation.indexOf("#page") > -1){
$('.pageLink[href='+ urlLocation +']').trigger('click')
}
});
function loadPage(newHash)
{
//This is the current Page
var curHash = $('#pageContent').data('currentPage');
//and this is the new one
newHash = newHash.replace('#page', '');
if(curHash===newHash){
//If already loaded: do nothing
return
}
$('#loading').css('visibility','visible');
$.ajax({
type: "POST",
url: "load_page.php",
data: 'page='+newHash,
dataType: "html",
success: function(msg){
if(parseInt(msg)!=0)
{
$('#pageContent').html(msg).data('currentPage',newHash);
$('#loading').css('visibility','hidden');
}
}
});
}
function loadPlay(newHash)
{//Similar to loadPage...
var curHash = $('#reproductor').data('currentPlay');
newHash = newHash.replace('#play', '');
if(curHash===newHash){return}
$('#loading').css('visibility','visible');
$.ajax({
type: "POST",
url: "load_play.php",
data: 'play='+newHash,
dataType: "html",
success: function(msg){
if(parseInt(msg)!=0)
{
$('#reproductor').html(msg).data('currentPlay',newHash);
$('#loading').css('visibility','hidden');
}
}
});
}
Check this and comment if this is what you need, or I got something wrong :)
There are a number of reasons why the following is not an ideal solution. The most glaring would be security - by modifying the href attribute of the link before clicking it, the user can certainly get your server to serve up any html on your server.
EDIT I've removed my original answer, because I can't recommend it's usage.
As Asad suggested, you can also use jQuery load and pass it the relevant url using some of the code above
function loadPage(url)
{
// remove the hash in url
url=url.replace('#','');
// extract page or play - only works for four letter words
var contentType=url.substr(0,4);
// extract the number
var contentId=url.substr(4);
if ( $contentType == "page") {
$("#pageContent #loading").css('visibility','visible');
$("#pageContent").load($contentType+'s/'+$contentType+'_'+$contentId+'.html');
$("#pageContent #loading").css('visibility','hidden');
} else if ( $contentType == "play") {
$("#reporductor #loading").css('visibility','visible');
$("#reproductor").load($contentType+'s/'+$contentType+'_'+$contentId+'.html');
$("#reporductor #loading").css('visibility','hidden');
}
}
I have a datatable that I am attempting to put drill-downs into based on the output of a php file, but I am having a few issues I can't seem to figure out. I am using http://datatables.net/blog/Drill-down_rows as a guide. So far this is my code:
Javascript:
<script type="text/javascript">
$("tr").live("click", function(){
var nTr = this;
var i = $.inArray(nTr, anOpen);
//oTable = my datatable
var oData = oTable.fnGetData(nTr);
if(i === -1) {
$(this).addClass('row_selected');
//THIS IS WHERE I AM GETTING A LITTLE LOST
//I WANT THE VALUE OF response.details TO BE STORED IN nDetailsRow
//oData.url is a mDataProp stored in the datatable row that contains the PHP link (this works okay)
var nDetailsRow = $.get(oData.url, function(response) {
//I don't really understand exactly what this is doing... but response.details is what I want to display from PHP
oTable.fnOpen(nTr, response.details, 'details');
});
//THIS LINE DOES NOT WORK CORRECTLY BECAUSE nDetailsRow IS NOT WHAT I WANT IT TO BE
$('div.innerDetails', nDetailsRow).slideDown();
anOpen.push(nTr);
else {
...
}
}
</script>
PHP:
<?PHP
$tableOut = '<div class="innerDetails">
<table cellpadding="5" cellspacing="0" border="0" style="padding-left:50px;">
<tr><td>Test Cell:</td><td>This is a test</td></tr>
</table>
</div>';
$data = array();
$data['details'] = $tableOut;
echo json_encode($data);
?>
I feel like I am almost there, but I don't quite understand the ajax $.get and am not sure if I am actually getting the JSON correctly from the PHP file. I also don't quite understand how to store that JSON in the active jquery code. Any ideas on how to accomplish these tasks and how to get my drill-down to display the PHP JSON data response.details?
I figured it out almost immediately after posting, I needed to use $.getJSON and include the last two lines inside that function. I am now using the following code which appears to work correctly:
<script type="text/javascript">
$(document).on("click", "tr", function(){
var nTr = this;
var i = $.inArray(nTr, anOpen);
//oTable = my datatable & oData.url is the mDataProp link to the PHP page
var oData = oTable.fnGetData(nTr);
if(i === -1) {
$(this).addClass('row_selected');
$.get(oData.url, function(response) {
oTable.fnOpen(nTr, response.details, 'details');
$('div.innerDetails', response.details).slideDown();
anOpen.push(nTr);
});
else {
...
}
}
</script>
Edit: Added/changed code from suggestions
<script type="text/javascript">
$("tr").live("click", function(){
var nTr = this;
var i = $.inArray(nTr, anOpen);
//oTable = my datatable
var oData = oTable.fnGetData(nTr);
if(i === -1) {
$(this).addClass('row_selected');
//THIS IS WHERE I AM GETTING A LITTLE LOST
//I WANT THE VALUE OF response.details TO BE STORED IN nDetailsRow
//oData.url is a mDataProp stored in the datatable row that contains the PHP link (this works okay)
var nDetailsRow = $.get(oData.url, function(response) {
try{
eval(response);
delete response;
//I don't really understand exactly what this is doing... but response.details is what I want to display from PHP
oTable.fnOpen(nTr, req.details, 'details');
}catch(e){
// error
}
});
//THIS LINE DOES NOT WORK CORRECTLY BECAUSE nDetailsRow IS NOT WHAT I WANT IT TO BE
$('div.innerDetails', nDetailsRow).slideDown();
anOpen.push(nTr);
else {
...
}
}
</script>
PHP:
var req = { details : '<?php echo str_replace('%', '\\x', urlencode('
<div class="innerDetails">
<table cellpadding="5" cellspacing="0" border="0" style="padding-left:50px;">
<tr>
<td>Test Cell:</td>
<td>This is a test</td>
</tr>
</table>
</div>')); ?>' };
I am struggeling to implement a function to open a new window with an external link. I am creating dynamic content and would like to write a function that opens up a pop/up window with the given link. Please help. I like jquery dialog boxes, and lightwindo etc. But am unable to implement any of these, any help would be gerat, am stuck and frustrated with this supposingly small issue . . . No option has worked for em so far. here the code of how i display my results, and how i think I have to call the function to open a new window. So the second function here is the one I would need some help with, to open a nice pop-up window displaying the information..
// --------------------- display the course results -> structuring the returned array, to output all information into a table ----------------------
function displayCourses()
{
var str = ' <table border="0" width="530">' +
'<tr>' +
'<td width="150">Title / course code</td>' +
'<td>INFO</td>' +
'</tr>';
if(curCourseList == null)
{
str = str + '<tr><td colspan="2"><div id="msgDips"></div></td></tr>';
}
else
{
for (var i = 0; i < curCourseList.length; i++)
{
str = str + '<tr><td valign="top" width="150"><a style="cursor:pointer;" onclick="showCourse(\''+curCourseList[i][0]+'\')" >' + curCourseList[i][0] + ' <br /> </a>' + curCourseList[i][1] +'<br>'+ curCourseList[i][3] +'<br /><br />'+ curCourseList[i][4] +'</td><td>' + curCourseList[i][2] +'</td></tr>';
}
}
str = str + '</table>';
document.getElementById("courseContainer").innerHTML = str;
if(curCourseList == null)
{
getLangToken('99');
}
}
function showCourse(code)
{
//alert(1)
$.ajax({
async:false,
type: "POST",
url: 'formPostsUser.php?reqtype=getCourse',
data:'coursecode='+ code,
success: function(data)
{
newwindow=window.open(url,'name','height=200,width=150');
if (window.focus) {newwindow.focus()}
return false;
}
});
}
And seriously any help will be greatly ap[preciated. am in such deep shit atm
EDIT---------------
It is the window.open bit i would liek to replace to open a nice fancy pop-up, not new browser window or page
I do not know WHY you say a dialog does not work or specifically if jQuery UI dialog does not work for you, but just in case:
$(function() {
$("<div id='helpDialog' style='display:none' />").appendTo(document);
$("#helpDialog").dialog({
autoOpen: false
});
$("#helpDialog").dialog('option', 'buttons', {
"Close": function() {
$(this).dialog("close");
}
})
});
function helpPopUp(page, height, width) {
$("#helpDialog").dialog('option', 'height', height);
$("#helpDialog").dialog('option', 'width', width);
$('#helpDialog').load(page);
$('#helpDialog').dialog("open");
}
helpPopUp('#helpDialog', 'Help/Summary.html', 550, 750);
You could of course, modify the .load(page) to be custom IF needed referencing the page in your url.
Here is a simple sample of the above with a slight mod to load your table instead of the page: http://jsfiddle.net/MarkSchultheiss/ABqrD/1/
If your triggering the window opening via anything other than user interaction it will most likely be blocked. Have you checked for notices pertaining to this?
If you don't want to use a real popup, you could try to create a <div> which 'hovers' above the page with something like
.hoverwindow {
position:absolute;
width: ...px;
height:...px;
margin:auto;
z-index:10; /*ensure that it's displayed on top*/
}
You could do this with a small modification of your function:
function showCourse(code)
{
$.ajax({
async:false,
type: "POST",
url: 'formPostsUser.php?reqtype=getCourse',
data:'coursecode='+ code,
success: function(data)
{
var newwindow = $('<div class="hoverwindow" />');
newwindow.html(data).appendTo('body');
}
});
}
Although frankly this is pretty much what plugins for "popups" do so all of this should be readily available online if you search for it. Hence why I have not bothered to actually try and test this code - it is simply to give an idea of what you could do.