I'm trying to read specific div-elements of a website with a script either written in php or perl.
Unfortunately, the page requests a login before those specific site can be read. As I can see, it's ssl-protected. I'm not looking for a complete solution, I just need a hint regarding the best way to tell the script the informations needed for logging in (user+password), before reading parts of the sourcecode of the page that comes afterwards.
I'm not quite sure if it's better to do this with PERL or PHP, so i have tagged this question with both of these languages.
Mojo::UserAgent (see cookbook) has a built-in cookie jar and can do SSL if you have IO::Socket::SSL installed. It has a DOM parser which can easily use CSS3 selectors to traverse the returned result. And if that wasn't good enough, the whole thing can be used non-blocking (if that's something you need).
Mojo::UserAgent and the other tools listed above are parts of the Mojolicious suite of tools. It's a Perl library, and I would certainly recommend Perl for this task since it is a more general purpose language than PHP is.
Here is a very simplistic example to get the text from all the links that are inside a div with a class myclass
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
$ua->post( 'http://mysite.com/login' => form => { ... } );
my #link_text =
$ua->get( 'http://mysite.com/protected/page' )
->res
->dom('div.myclass a')
->text
->each;
In fact, running this shell command may be enough to get you started (depending on permissions)
curl -L cpanmin.us | perl - -n Mojolicious IO::Socket::SSL
Related
So I was wondering if you could use F# with fsi.exe to run server side scripts to serve html pages. Basically, could you use it like php? If you can, would it be very practical to use it like that? Also, even if this is not very practical, I would be interested in if this would somehow be possible.
PHP and for that matter Classic ASP pages on a web site typically map one-to-one to script files on the file system on a web server.
If you implement ASP.Net's IHttpHandler you could execute F# scripts on the file system in response to HTTP requests. The IHttpHandler's ProcessRequest method passes a HttpContext instance which could be used for reading the Request arguments and setting the Response. This could be passed to a well-known function implemented, say "handler", in the F# script.
type HttpHandler() =
interface IHttpHandler with
member this.ProcessRequest(context:HttpContext) =
let localpath = context.Server.MapPath(context.Request.FilePath)
let a = compile localpath // where compile creates an assembly from an fsx file
let mi = getHandler a // where getHandler gets a function using reflection
mi.Invoke(null, [|context|])
The F# Compiler Services provides FSharperChecker API which could be used to compile F# script files to dynamic assemblies.
let compile path =
let checker = FSharpChecker.Create()
let errors, exitCode, dynamicAssembly =
checker.CompileToDynamicAssembly([| "-o"; path; "-a"; path |], execute=None)
|> Async.RunSynchronously
dynamicAssembly
The dynamic assemblies could then be used to invoke functions via reflection, passing the HttpContext value.
let getHandler a =
a.GetTypes() |> Seq.pick (fun ty ->
match ty.GetMethod("handler", [|typeof<System.Web.HttpContext>|]) with
| null -> None
| mi -> Some mi
)
So that your script file might look something like this:
let handler (context:System.Web.HttpContext) =
context.Response.Write("Hello World")
Finally there are a plethora of F# DSLS for generating HTML from code, I have a simple one called FsHtml.
Giraffe with the Razor view engine might be the best fit for what you're after: https://github.com/giraffe-fsharp/Giraffe.Razor
It provides a nice way to define API routes functionally and a nice clean syntax for HTML templates.
It does use asp.net core though - I'm not sure what you're trying to achieve by avoiding this so apologies if my answer is a little off-target.
You can compile .fsx scripts with Fable (see here: https://axxes.com/en/net/compiling-f-scripts-with-fable-2-2/) as Just another metaprogrammer has mentioned, though personally it's not my preferred approach (I'm happy with .fs) and I'd argue that it's more akin to react / jsx since it's client side and uses react under the hood.
I'm not entirely sure the wording for the title is correct, but what I'm attempting to do is run and execute PHP files from within the Lift framework.
I'm not after any url queries to a PHP file residing on a server, more interested in somehow getting the PHP runtime working through my Scala/Lift app.
Use case: I have my app packaged into a .war file, I host this via a cloud provider. I upload code snippets to said app which then runs the php file and does whatever necessary.
I've seen various posts regarding Bianca but am hoping to keep this setup light and require only the PHP binary itself and a little code to get it flying.
Thanks in advance, please let me know if you need me to elaborate :)
“Never say never, because limits, like fears, are often just an
illusion.”
― Michael Jordan
What you really need is an open source (GPL), embeddable, full PHP 5 implementation, written entirely in Java!
Caucho's Quercus PHP Java runtime is just that, and it will let you run PHP within a Java app without external libraries or native code.
Below is a Quercus-PHP-in-Java code sample I found in this answer
import javax.script.ScriptEngine;
import com.caucho.quercus.script.QuercusScriptEngineFactory;
QuercusScriptEngineFactory factory = new QuercusScriptEngineFactory();
ScriptEngine engine = factory.getScriptEngine();
String phpCode = "<?php $foo = strlen('abc'); print $foo; return 'yikes'; ?>"; //PHP Code as String
Object o = engine.eval(phpCode);
System.out.println(o);
It should be little effort to convert this code to idiomatic Scala. Obviously, the 'phpCode' variable could be constructed from external PHP file contents etc.
Let us know how you get on ;-)
That's a bit of an odd requirement, but if it's what you need to do, you can use a ProcessBuilder to execute and interact with your PHP script from the command line.
This isn't for anything serious, more so to just play around and experiment. Just not sure where I would start. Also not bound to Node.js, just figured it's the most popular server-side JS tool around.
Though if there are some major technical reasons why one shouldn't do something like this on a real site, feel free to throw them out there.
What I'm asking about is something like…
$input = $js_string_built_via_php;
if ($run_on_server)
$output = node.js($input);
else
$output = '<script type="text/javascript">'.$input.'</script>';
echo $output;
There are several alternatives to using node here.
Have you seen the V8js PHP extension, which integrates the V8 engine with PHP? I don't have any experience with using it myself, though.
Alternatively, similar to using node, you could install Rhino (available in the Ubuntu repos, at least from 12.04). Or another command line javascript interpreter. Then you can execute javascript from the shell:
rhino -e '<javascript code>';
So you should be able to do something like the following. You would have to use print to output data and then parse the output back in php somehow:
<?php
function execute_js($script) {
return shell_exec('rhino -e '.escapeshellarg($script));
}
$javascript = "
function add(a,b) {
return a+b;
}
print(add(5,6));
";
$result = execute_js($javascript);
print $result;
I doubt this would be a good idea in a production application and seems like it might be quite vulnerable with a much greater attack surface. With clients being able to inject javascript code that actually gets executed on the server. It's probably very slow at load also.
A better solution 3 years later?
dnode-php would a better solution for what you want today.
There is a cool introduction here. As mentioned by the author Henri Bergius:
DNode is a remote method invocation protocol originally written for
Node.js, as the name probably tells. But as the protocol itself is
quite simple, just sending newline-terminated JSON packets over TCP
connections, implementations have started popping up in other
languages. You can talk DNode in Ruby, Perl, Python, Java, and now
PHP.
I need to compare a webpage's DOM structure at various points in point. What are the ways to retrieve and snapshot it.
I need the DOM on server-side for processing.
I basically need to track structural changes to a webpage. Such as removing of a div tag, or inserting a p tag. Changing data (innerHTML) on those tags should not be seen as a difference.
$html_page = file_get_contents("http://awesomesite.com");
$html_dom = new DOMDocument();
$html_dom->loadHTML($html_page);
That uses PHP DOM. Very simple and actually a bit fun to use. Reference
EDIT: After clarification, a better answer lies here.
Perform the following steps on server-side:
Retrieve a snapshot of the webpage via HTTP GET
Save consecutive snapshots of a page with different names for later comparison
Compare the files with an HTML-aware diff tool (see HtmlDiff tool listing page on ESW wiki).
As a proof-of-concept example with Linux shell, you can perform this comparison as follows:
wget --output-document=snapshot1.html http://example.com/
wget --output-document=snapshot2.html http://example.com/
diff snapshot1.html snapshot2.html
You can of course wrap up these commands into a server-side program or a script.
For PHP, I would suggest you to take a look at daisydiff-php. It readily provides a PHP class that enables you to easily create an HTML-aware diff tool. Example:
<?
require_once('HTMLDiff.php');
$file1 = file_get_contents('snapshot1.html');
$file2 = file_get_contents('snapshot1.html');
HTMLDiffer->htmlDiffer( $file1, $file2 );
?>
Note that with file_get_contents, you can also retrieve data from a given URL as well.
Note that DaisyDiff itself is very fine tool for visualisation of structural changes as well.
If you use firefox, firebug lets you view the DOM structure of any web page.
i have two pages one in php(index.php) and another one in Perl(dbcon.pl).
basically i want my php file to show only the UI and all the data operations would be done in Perl file.
i have tried
in index.pl
<?php include("dbcon.pl");?>
<html>
<br/>PHP</br>
</html>
and dbcon.pl has
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use CGI::Simple;
my $cgi = CGI::Simple->new;
my $dsn = sprintf('DBI:mysql:database=%s;host=%s','dbname','localhost');
my $dbh = DBI->connect($dsn,root =>'',{AutoCommit => 0,RaisError=> 0});
my $sql= "SELECT * FROM products";
my $sth =$dbh->prepare($sql);
$sth->execute or die "SQL Error: $DBI::errstr\n";
while (my #row = $sth->fetchrow_array){
print $cgi->header, <<html;
<div> #row[0] #row[1] #row[2] #row[3] #row[4]</div>
html
}
but when i run index.php in browser it prints all the code in dbcon.pl file instead of executing it
how to overcome this problem?
note: i am running this in windows environment
is there any other way to do this?
May I ask what the problem really is? I don't see anything "special" in the Perl code, so you either:
a) Don't know how to access your DB from PHP (i.e. you don't know PHP) or
b) Don't know what Perl is doing (i.e. you don't know Perl) or
c) possibly your environment is set up so that you can use Perl DBI but you can't do the same from PHP.
This link should give you pointers to do what you are doing in Perl directly from PHP. You will easily find dozens of examples for various PHP/DB combinations.
The only other way would be to do what another poster suggests: invoke the Perl script and parse the result (printed to standard out).
This is rubygoldbergsque, brittle and unacceptable as a solution unless you are absolutely desperate to use something that is available only as a Perl module (which is not the case from the example you posted).
In general if you want to have something done in a language and use it from some other language the best way would be to make the (in your case) Perl run as a sort of "server", i.e. a seperate process - and make it expose services using XML-RPC or some other lightweight protocol.
INVOKING PROGRAMS WITH exec() OR SIMILAR CONSTRUCTS IS EXTREMELY BAD PRACTICE.
What you are trying is not possible that easy. You will have to execute the perl script with PHP, capture the output and print it like:
<?php echo exec('perl dbcon.pl'); ?>
As mentioned that is not a good thing to do. For a good separation between backend and user interface you should have a look at existing PHP frameworks.
There is Perl PECL package to integrate Perl into PHP.
P.S. IMHO it is better to use templating system like Template Toolkit in Perl. You can even use Perl inside templates.
If you're using Catalyst you could us Catalyst::View::PHP I suspect it will give you more clues on how to use php as your templating system. It also mentions PHP::Interpreter