I am building a scraper to scrape content using guzzle and symfony dom crawler But I run into an issue.
The page I am scraping has multiple Iframe servers They default iframe is shown when the scraper loads the page but in order to get the other servers it needs to click there buttons and so it reflects the server iframe.
How do I do that?
Guzzle can't render Javascript websites so it doesn't support that
There are two ways around this either check for network tabs and send the request to the ajax link they get the iframes from, or use symfony Panther as headless browser library they support javascript websites and even Bypass Cloudflare Protection
Related
Is this possible to take screenshots of windows's screen of client users using a PHP application?
If you want to capture the current client web page, you should look into JS solutions (see this related SO answer):
html2canvas
npm packages with keyword ‘screenshot’
You may also be able to programatically load and render HTML with a GET request, though you will still need to run JavaScript to render the full web page on the server side.
If you mean a screenshot of the actual desktop, I fear only the browser would be able to do it and this is still at an experimental stage, probably due to security concerns:
Chrome desktopCapture extension
MDN Media Capture and Streams API
Can I use media capture?
Edit: Possible duplicate of this question. Some back-end libraries to render a web page:
dompdf (PHP5)
wkhtmltopdf (C++)
I want to crawl https://www.socialbakers.com/ but I am having trouble handling the multiple redirections when doing a post in logging in to the site. First redirect is to a HTML page with JavaScript doing the redirection.
I want to trace all the redirection request URL and if it is possible to do post/get request to each URL.
Goutte is a simple web scraper, because it doesn't support JS.
If you are stuck with JS, then you need something more powerful, like Selenium or PhantomJS.
Take a look at PhantomJS, it's simple and fast.
Selenium also has a driver for PhantomJS, if you prefer to stay with PHP (you can use Mink with Selenium and PhantomJS).
I am building an app and because of my missing knowledge of java and Xcode I decided to make this app in HTML,PHP,CSS & javascript(jquery). Now I'm facing the problem of getting this app to the different devices and I want to use PhoneGap for that. I placed an index.html inside the server for the app and there is an iframe placed inside and nothing more. Is it still possible for me to get access to the media storage and send PHPdata back and forth while having an iframe as a bridge between PhoneGap on android and my website on its web server? Can I access the camera of the device?
The short answer: No, and you shouldn't try that.
The long answer (And the safer one) is to use postMessage as described in MDN, do it in a different way could cause inconsistency between devices (Remember how fragmented is android and how painful could be the backward compatibility with 4.3 and below)
So, you could get the iFrame element and then post a message like
otherWindow.postMessage(InfoToSend, "*");
In the same way, you could listen to that event inside the frame:
window.addEventListener("message", receiveMessage, false);
This will no cause cross-frame issues and it will be the safer way to pass information between the frames, this work in both directions, so you could ask (From inside your iFrame) to use some native feature (plugin), listen to this event in your device main app, executed what you need and after that send a postMessage (from your app) to your iFrame with the result (a base64 img for example)
If you are trying to pass the window.cordova the instance or something similar, it won't work, so you will need to establish a conversation between the iFrame and the window.top frame.
I am new in PhoneGap,
I have a Joomla Site, I want to create a PhoneGap App using that Joomla site. so is this possible ? if yes than how can i implement it ?.
You can use. But you have to create web services for request and responses. Because Phonegap will not run PHP script. So you can create webservices to access Joomla functionality via JSON responses from your joomla. You can get the JSON data via jQuery in your phonegap app.
and someone said PGBuild will convert webpage into phonegap app. You can try it also.
I have succeeded creating an application as follows:
Design your Joomla page so it's responsive and renders correctly on your mobile;
Download the whole thing with wget
Edit all urls and make them point to the right (local-remote) locations.
I saved all css, images, js locally so the page would load even offline; then changed all the links so they point to the remote host;
All forms should point to the remote host;
Most modules and components were changed so after the initial display (local) they will load the updated info in the frame.
Popups won't be available, so you'll need heavy workarounds for facebook integration etc.
Also, if you plan to use any phonegap features, they will only be available if you load the library inside your downloaded page; this will force you to change all calls to ajax (no document.location.href calls are possible since you'll lose the phonegap javascript).
The first time you load the page from the server it will take forever to load. Ensure you have proper caching set up otherwise you'll just lose your customers.
My advice is to start with an ajax project at the very beginning, it will save you lots of headaches.
Keep in mind, if you plan to use iframes, forget it. They work lousily on iphones, and debugging is nearly impossible. Simply take your time to plan a real phone app in javascript & using Ajax for the calls.
You don't need any webservices in Joomla for loading content, simply use:
&tmpl=component
when you need the content of a module.
Additionally, for blogs and lists, Joomla makes RSS feeds available so you shouldn't need too much coding to achieve this.
I have limited myself to ajaxifying the modules, which makes caching more efficient on the server and the interfaces appear more dynamic.
If I have an iframe loading a website from a different server, like bing.com, can I manipulate or retrieve the code from that iframe?
If not, I can do a curl request and create the iframe from that and then manipulate it, right?
In javascript you can't access the contents of an iframe if that site resides on a different domain than the one you are loading the iframe from. You can proxy the page load by having a PHP scrip that "curls" the request. There is a lot more involved than that though to cleanly proxy another site.