I noticed that my browser keeps different cookies than my curl cookie files:
"__utma=256586655.848991821.1337158982.1337158982.1337179787.2; __utmz=256586655.1337179787.2.2.utmcsr=login.example.com|utmccn=(referral)|utmcmd=referral|utmcct=/company.php; __utmc=256586655; PHPSESSID=8sedo85uc5rfpnluh06bdb0mk4"
And this is my curl based cookie.txt:
# Netscape HTTP Cookie File
# http://www.netscape.com/newsref/std/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.
login.example.com FALSE / FALSE 0 PHPSESSID 8peqektoc5j3570h08efa6o3n2
So, how to create utma utmz values and what is that values stand for ?
Those are Google Analytics cookies. Possibly, you're not telling curl to download third-party scripts referenced by the page.
Try wget with --page-requisites and --save-cookies and --load-cookies. It will download files used by the page, such as scripts.
Unfortunately, it still might not load Analytics if it's using the typical script-driven async loader, since JavaScript has to actually execute for that to work.
Those are Google Analytics cookies. Unless you pull & run those scripts (faking the Javascript runtime environment of a browser), you won't get them. Fortunately, they're irrelevant to anything you actually want to do.
Related
After recording script, I have added 'Cookie Manager'.
2. While running, Cookies are not showed for Request headers in Jmeter and the Connection closed error showing for listener.
But, Browser cookies are passing the request headers in my application.
So, How can i pass cookies in jmeter. kindly give me a solution.
Please refer the snapshot.
Thanks,
Vairamuthu.
Just add to Test Plan an HTTP Cookie Manager before your request.
I don't think you should be sending cookie. I would rather expect that you need to extract the value of this _xsrf cookie from the /api/login request and add the extracted value as X-Xsrftoken header.
Add the next line to user.properties file (located in JMeter's "bin" folder"
CookieManager.save.cookies=true
and just in case this line as well:
CookieManager.check.cookies=false
Restart JMeter to pick the properties up
Add HTTP Header Manager to the request which is failing
Make sure it contains the following like:
Name: X-Xsrftoken
Value: ${COOKIE__xsrf}
More information:
Configuring JMeter
How to Load Test CSRF-Protected Web Sites
First of all, the purpose of this is to spider one of our signed-in applications and get data about jobs run which I may not be able to get any other way.
I can log in via a browser, and also can inspect my cookies; how would I then take that (in a timely manner) and add that information to a cURL call so that I can use PHP to parse the return page (and links)?
cURL can read cookies from a Netscape cookie format file.
Since you can inspect the cookies, use this Netscape format cookie file generator to convert them to a file.
Then use curl_setopt($ch, CURLOPT_COOKIEFILE, 'file.txt'); so cURL will read the cookies from there where file.txt is the file you saved the cookies to.
I think my question seems pretty casual but bear with me as it gets interesting (at least for me :)).
Consider a PHP page that its purpose is to read a requested file from filesystem and echo it as the response. Now the question is how to enable cache for this page? The thing to point out is that the files can be pretty huge and enabling the cache is to save the client from downloading the same content again and again.
The ideal strategy would be using the "If-None-Match" request header and "ETag" response header in order to implement a reverse proxy cache system. Even though I know this far, I'm not sure if this is possible or what should I return as response in order to implement this technique!
Serving huge or many auxiliary files with PHP is not exactly what it's made for.
Instead, look at X-accel for nginx, X-Sendfile for Lighttpd or mod_xsendfile for Apache.
The initial request gets handled by PHP, but once the download file has been determined it sets a few headers to indicate that the server should handle the file sending, after which the PHP process is freed up to serve something else.
You can then use the web server to configure the caching for you.
Static generated content
If your content is generated from PHP and particularly expensive to create, you could write the output to a local file and apply the above method again.
If you can't write to a local file or don't want to, you can use HTTP response headers to control caching:
Expires: <absolute date in the future>
Cache-Control: public, max-age=<relative time in seconds since request>
This will cause clients to cache the page contents until it expires or when a user forces a page reload (e.g. press F5).
Dynamic generated content
For dynamic content you want the browser to ping you every time, but only send the page contents if there's something new. You can accomplish this by setting a few other response headers:
ETag: <hash of the contents>
Last-Modified: <absolute date of last contents change>
When the browser pings your script again, they will add the following request headers respectively:
If-None-Match: <hash of the contents that you sent last time>
If-Modified-Since: <absolute date of last contents change>
The ETag is mostly used to reduce network traffic as in some cases, to know the contents hash, you first have to calculate it.
The Last-Modified is the easiest to apply if you have local file caches (files have a modification date). A simple condition makes it work:
if (!file_exists('cache.txt') ||
filemtime('cache.txt') > strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE'])) {
// update cache file and send back contents as usual (+ cache headers)
} else {
header('HTTP/1.0 304 Not modified');
}
If you can't do file caches, you can still use ETag to determine whether the contents have changed meanwhile.
I will have an app where I will prompt users for a URL (with proper regex url validation) and return the page with cURL and run some checks on it.
What would be the most secure way of returning a remote webpage securely with cURL? As I understand even cURL has some vulnerabilities, like 'safe mode' Security Bypass (http://www.securityfocus.com/bid/27413).
SecurityFocus claims this has been fixed in PHP 5.2.6 . If you can't upgrade to that, you need to manually check for that attack vector. Perhaps check in your user input if the url definitely has "http" in front of it, with if (substr($url, 0, 7) == 'http://'))
Furthermore, according to the comments on this php bug report curl gives you the option to disable specific protocls, including local file access, but only when you configure and compile from source. According to the cURL install manual it must be something like this (untested):
./configure --disable-file
I'm attempting to use curl inside php to grab a page from my own web server. The page is pretty simple, just has some plain text output. However, it returns 'null'. I can successfully retrieve other pages on other domains and on my own server with it. I can see it in the browser just fine, and I can grab it with command line wget just fine, it's just that when I try to grab that one particular page with curl, it simply comes up null. We can't use file_get_contents because our host has it disabled.
Why in the world would this be different behavior be happening?
Found the issue. I was putting my url someplace that was not in curl_init(), and that place was truncating the query string. Once I moved it back to curl_init, it worked.
Try setting curl's user agent. Sometimes hosts will block "bots" by blocking things like wget or curl - but usually they do this just by examining the user agent.
You should check the output of curl_error() and also take a look at your logfiles for the http server.