I have researched many python post and sites to build a few other scripts that log into specific accounts of mine to scrape/parse data from a site. I have come across one I cannot figure out, relayrides.com. The site uses PHP and I have successfully logged into other php based pages using python, but it seems a JSESSIONID and rr_u_cid are set when the pages loads and passed with the POST from, see sample code used to login to other pages:
`payload = {
'action': '/login',
'email': 'user#gmail.com',
'password': 'pass1234'
}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36'
}
with session() as c:
c.post('https://relayrides.com/login', data=payload)
response = c.get('https://relayrides.com/earnings', headers=headers)
print(response.text)`
I've tried pass the settings a cookie in the header, using CookiJar, etc. I am just new to python and web-scraping with python.
Thanks,
dickie
Related
I am trying to figure out how can I call php functions running on remote of shoretel IP phone .
I can access it through browser by simple IP and it is flash player script but I want to do few things programmatically and I don't find any help to do the same.
The flash player app is calling following api
https:///gateway.php?PHPSESSID=6432fc5f0999d0f4536fced79b95
and in payload of this it is calling php function like LSRoom_Remoting.getHeaderInfo
I am trying but getting error like "Malformed AMF message"
Here is the code
static void Main(string[] args)
{
ServicePointManager.ServerCertificateValidationCallback += (o, c, ch, er) => true;
var content = new StringContent("LSRoom_Remoting.getBoardID", Encoding.UTF8, "application/x-amf");
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Add("Accept","*/*");
client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.9");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
//client.DefaultRequestHeaders.Add("Content-Length", "47");
//client.DefaultRequestHeaders.Add("Content-Type", "application/x-amf");
client.DefaultRequestHeaders.Add("Cookie", "PHPSESSID=3cbc7ff268e80e78e1458a4adfdd1f56");
client.DefaultRequestHeaders.Add("DNT", "1");
client.DefaultRequestHeaders.Add("Host", "10.6.32.21");
client.DefaultRequestHeaders.Add("Referer", "https://10.6.32.21/interface/ls_interface.swf");
client.DefaultRequestHeaders.Add("Origin", "https://10.6.32.21");
client.DefaultRequestHeaders.Add("X-Requested-With", "ShockwaveFlash/32.0.0.114");
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");
var result = client.PostAsync(string.Format("https://10.6.32.21/gateway.php?PHPSESSID={0}", "3cbc7ff268e80e78e1458a4adfdd1f56"), content).Result;
//var result = client.DownloadString("https://10.6.32.21/interface/interface.php?uniqueKey=152322314997139&brand=shoretel&key=");
var obj = result.Content.ReadAsStringAsync();
Console.WriteLine(result.Content.ToString());
Console.ReadKey();
}
}
What i am doing wrong?
it seems likely, that the PHPSESSID is long expired, while not using one from a recent request; because the default timeout is 15 minutes. in case it had not yet been expired, you still have to build a proper AMF message, which consist of a header and a body. there is a library for that: amfphp, which at least should provide some clue how it should look alike, despite it is written in PHP.
I'm trying to login into a website and send an sms with python. I'm using the request module, passing the user name and password. After I have passed that I am unable to judge whether the login has succeeded or not.
url='http://www.connectexpress.in'
headers={"User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"}
s=requests.session() ; s.headers.update(headers)
r=s.get(url) ; soup=BeautifulSoup.BeautifulSoup(r.content)
VIEWSTATE=soup.find(id="__VIEWSTATE")['value']
#VIEWSTATEGENERATOR=soup.find(id="__VIEWSTATEGENERATOR")['value']
EVENTVALIDATION=soup.find(id="__EVENTVALIDATION")['value']
login_data={"__VIEWSTATE":VIEWSTATE,
"__EVENTVALIDATION":EVENTVALIDATION,
"txtUserName":usrNme,
"txtPassword":paswrd,
"submit":'Submit'}
r1=s.post(url, data=login_data)
This is the piece of code that I used and could any one give some ideas ?
I need to get requesting browsers name in my web app.( for analytics )
In core php when I use $visitor_user_agent=$_SERVER['HTTP_USER_AGENT']it returns Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36 string when using with chrome.And then preg_match('/Chrome/i', $visitor_user_agent) can be used to know if its chrome or not.I am not sure if that was efficient way to find browser name or not.
I also found get_browser link but it is not giving browser name.
Is there a way in cakephp3 or core php to get browser name ?
This would return the user agent used for the request:
$this->request->header('User-Agent');
http://book.cakephp.org/3.0/en/controllers/request-response.html
Look into documentation of Request object.
You can get HTTP_USER_AGENT using env() method:
$this->request->env('HTTP_USER_AGENT');
You can also prepare custom detector:
$this->request->addDetector(
'chrome',
['env' => 'HTTP_USER_AGENT', 'pattern' => '/Chrome/i']
);
And then in controller just use is() method as follows:
if($this->request->is('chrome')) {
// do stuff for chrome
}
I want to determine which user uses which browser when they visit to my website.
For this process i'am using $_SERVER['HTTP_USER_AGENT'] but when i try to print it while i'am using chrome i get this result(on localhost):
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
What can i do to fix that ? Any help would be appriciated
$_SERVER['HTTP_USER_AGENT']
This gives the information of the browser as you said.
Always keep in mind that HTTP_USER_AGENT can be easily spoofed by the user.
get_browser() gives the browser's capability.
you can also use the php class provided by Wolfcast
use like:
$browser = new BrowserDetection();
echo 'You are using ', $browser->getBrowser(), ' version ', $browser->getVersion();
if you really want to differentiate the browsers from the client-side you could use the javascript to find the browser and apply the changes through that based on different browser.
this is a code that i obtained from internet that exactly differentiate the browsers.
var isOpera = !!window.opera || navigator.userAgent.indexOf(' OPR/') >= 0;
var isFirefox = typeof InstallTrigger !== 'undefined'; // Firefox 1.0+
var isChrome = !!window.chrome && !isOpera; // Chrome 1+
if(isFirefox){
//do the change
} else if (isChrome){
//do the change
}
Im looking to book a hotel with expedia api
https://book.api.ean.com/
but the url is always blank,
details here using rest/json
http://developer.ean.com/docs/hotels/version_3/book_reservation/
I've seen a few people experiencing the same problem and wondered if anyone knew what it was
https://book.api.ean.com/ean-services/rs/hotel/v3/res?cid=55505
&apiKey=xxx
&locale=en_US
¤cyCode=USD
&customerUserAgent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.83 Safari/535.11
&customerIpAddress=127.0.0.1
&room1=1,3
&room1FirstName=TestBooking
&room1LastName=TestBooking
&room1BedTypeId=13
&room1SmokingPreference=NS
&room2=1,5
&room2FirstName=TestBooking
&room2LastName=TestBooking
&room2BedTypeId=13
&room2SmokingPreference=NS
&email=xxx
&firstName=TestBooking
&lastName=TestBooking
&homePhone=TestBooking
&workPhone=TestBooking
&creditCardType=CA
&creditCardNumber=5401999999999999
&creditCardIdentifier=123
&creditCardExpirationMonth=11
&creditCardExpirationYear=2012
&address1=travelnow
&city=Bellevue
&stateProvinceCode=WA
&countryCode=US
&postalCode=98004
&customerSessionId=0ABAA871-3127-A913-6642-A1F86D902E2B
&hotelId=211540
&arrivalDate=12/10/2012
&departureDate=12/12/2012
&supplierType=E
&rateKey=d03a8d29-1df2-4436-81d6-6b37eb4dcb78
&roomTypeCode=352749
&rateCode=1279169
&chargeableRate=803.04
including or excluding the minorrev doesn't seem to make much difference
Try urlencoding each querystring parameter:
customerUserAgent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.83 Safari/535.11
arrivalDate=12/10/2012
The customerUserAgent url param should be url encoded:
customerUserAgent=Mozilla%2F5.0%20(Windows%20NT%206.1)%20AppleWebKit%2F535.11%20(KHTML%2C%20like%20Gecko)%20Chrome%2F17.0.963.83%20Safari%2F535.11
arrivalDate=12%2F10%2F2012
Take a look at PHP's urlencode for more information.
Also, be sure to remove the spaces between each querystring param, not sure if it was copy/paste issue or not.