Accessing a web site via web scrape - php

Question

When attempting to web scrape Rubies, I am unable to get past the login. I have absolutely no idea why I am not able to, but here are the cURL options that I am using. If anyone sees a problem, I would greatly appreciate it!
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.rubies.com/customer/account/loginPost/",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HEADER => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => array('form_key' => "****", "login[username]" => "****", "login[password]" => "****", "persistent_remember_me" => 'on', "send" => ''),
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_COOKIEFILE => 'cookie.txt',
CURLOPT_COOKIEJAR => 'cookie.txt',
CURLOPT_HTTPHEADER => array(
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Host: www.rubies.com',
'Content-Type: application/x-www-form-urlencoded',
'Origin: https://www.rubies.com',
'Referer: https://www.rubies.com/customer/account/',
'Connection: keep-alive',
'Cache-Control: no-cache',
'Upgrade-Insecure-Requests: 1'
),
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLINFO_HEADER_OUT => true
));
I currently have the form key hard encoded, but I am not sure if I would have to change the form key depending on the login. The response from the post is empty, but I get redirected 2 times. Once to the account page, then back to the login. If anyone can tell me what is going on, then I would appreciate it. I think they are using some kind of basic auth system.

Answer 1

Use fiddler2 or another packet sniffer to look at the cURL traffic both requests and responses. Compare that to the traffic using a browser.
You probably either missed or mistyped a field, or missed follow-up steps like setting cookies and posting additional data.
Code for a login often requires fetching the login page, scraping a one-time token (changes with each page request), then posting as the first step. This might trigger script code to set cookies and/or automatically submit other data.

Answer 2

you do several mistakes.
you say to the server that your POST body is application/x-www-form-urlencoded encoded, but you give CURLOPT_POSTFIELDS an array, so what you actually send to the server, is multipart/form-data encoded. to have curl send the post data as application/x-www-form-urlencoded, urlencode the data for CURLOPT_POSTFIELDS - with arrays specifically, http_build_query will do this for you. furthermore, with POSTs when doing multipart/form-data or application/x-www-form-urlencoded, don't set the content-type header at all, curl will do it for you, automatically, depending on which encoding was used. on that note, you shouldn't set the User-Agent header manually, either, but use CURLOPT_USERAGENT. and you should not set the Host header either, curl generates that automatically, and you're more likely than curl to make a mistake.
also, here you send a fake Referer header, some websites can detect when the referer is fake, it's safer just to set CURLOPT_AUTOREFERER, and make a real request, thus obtaining a real referer. also, to actually login to https://www.rubies.com/customer/account/loginPost/ , you need both a cookie session, and a form_key code, the form_key is probably tied to your cookie session, and probably a form of CSRF token, but you provide no code to acquire either. and on top of that, you may need a real referer.
using hhb_curl from https://github.com/divinity76/hhb_.inc.php/blob/master/hhb_.inc.php ,
here's an example code i think would be able to log in, with a real username/password, doing none of the mistakes i listed above:
<?php
declare(strict_types = 1);
require_once ('hhb_.inc.php');
$hc = new hhb_curl ();
$hc->_setComfortableOptions ();
$hc->exec ( 'https://www.rubies.com/customer/account/login/' ); // << getting a referer, form_key (csrf token?), and a session.
$domd = #DOMDocument::loadHTML ( $hc->getResponseBody () );
$csrf = NULL;
// extract the form_key
foreach ( $domd->getElementsByTagName ( "form" ) as $form ) {
if ($form->getAttribute ( "class" ) !== 'form form-login') {
continue;
}
foreach ( $form->getElementsByTagName ( "input" ) as $input ) {
if ($input->getAttribute ( "name" ) !== 'form_key') {
continue;
}
$csrf = $input->getAttribute ( "value" );
break;
}
break;
}
if ($csrf === NULL) {
throw new \RuntimeException ( 'failed to extract the form_key token!' );
}
$hc->setopt_array ( array (
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query ( array (
'form_key' => $csrf,
'login' => array (
'username' => '???',
'password' => '???'
),
'persistent_remember_me' => 'on',
'send' => '' // ??
) )
) );
$hc->exec ( 'https://www.rubies.com/customer/account/login/' );
hhb_var_dump ( $hc->getStdErr (), $hc->getResponseBody () );
EDIT: fixed an url, the original code definitely wouldn't work, but it should now.