I'll explain the setup quickly, we have multiple domains, running on 2 servers (Dev and Live), which all are populated from a CMS database on the Live server. I'm adding in 404 reporting, so each site logs any 404's it gets, and we can view them in the CMS.
Each site, when our Framework detects a 404 error, it does a curl call to http://cms.example.com/log404.php and sends the $_SERVER variable. At the top of the log404.php I have this code which wraps the whole logging code.
if (in_array($_SERVER['REMOTE_ADDR'], array('dev server ip', 'live server ip'))) {
Then in here I store the relevant bits of data from $_POST. The reason I did it this way, rather than each site just directly adding to the database, was if we want to change the logging code somehow (log different data, write it a file, change the database etc), it only needs changing once, rather than in 15+ different sites.
Is the above if statement a safe way to check if the data was posted by us, and not somebody else? Or would it be possible for somebody to manipulate the curl call so the REMOTE_ADDR appears to be one of our IP's?
$_SERVER['REMOTE_ADDR'] uses the IP address from the TCP handshake so the same answer as this applies: Is it possible to pass TCP handshake with spoofed IP address?.
So if this is over the internet, then you are safe from the IP being spoofed.
However, for extra protection you should also protect your log service with authentication (as #moonwave99 suggested) and you should only run your log service over a HTTPS connection.
Related
I'm building a web app (example: www.mywebapp.example) that allows users to point their domain - www.xyz.example - to www.mywebapp.example. When users go to www.xyz.example, their content will be served from www.mywebapp.example instead. Users will be told how to update their # and www A records in their domain providers DNS settings to connect www.xyz.example to www.mywebapp.example.
I can manually create new SSL certificates using ./certbot-auto -d <domain-name> for each domain. I have also set a cron job to test for renewal.
However, I want to automate this process by running a PHP script, triggered from a JavaScript function, each time a user connects their domain to www.mywebapp.example. My questions are:
Should I execute the ./certbot-auto command from PHP using the exec()/shell_exec() command? Should I write a separate bash script and run the bash script instead?
Should I use an ACME PHP library recommended by LetsEncrypt - https://letsencrypt.org/docs/client-options/
I manually created a new SSL certificate for a domain www.xyz2.example, which successfully pointed to www.mywebapp.example. However, this broke SSL support for all existing domains - *.mywebapp.example, mywebapp.example, www.xyz.example. Do I need to create virtual hosts for each domain pointing to www.mywebapp.example?
Do I need to edit /etc/httpd/conf.d/ssl.conf to add the new virtual hosts? Can multiple domains use the same DocumentRoot path?
I've read through all the following links, but am still pretty confused:
https://httpd.apache.org/docs/2.4/vhosts/name-based.html
https://serverfault.com/questions/7308/dynamic-virtual-hosts-in-apache
https://serverfault.com/questions/126554/multiple-domains-with-ssl-on-same-ip
certbot-auto / letsencrypt setting up one key for multiple domains pointing to the same server
Any help is greatly appreciated. If more information is required, please let me know.
My server setup is:
LAMP
AWS EC2
You have many questions in one.
Should I execute the ./certbot-auto command from PHP using the exec()/shell_exec() command? Should I write a separate bash script and run the bash script instead?
In a generic case: no.
PHP is executed once a webpage has been reached, hence it lives inside the current HTTP session, and the browser at the other end will wait for some kind of response back, in some kind of limited time (and users get angry if they do not see something appearing "fast").
If you exec things you have two options:
you wait for completion of external program: problem, this can be after an "undefinite" time, so if this is not taken into account, user will never see anything back in their browser
you start it in the background, and you do not wait for its completion: problem, you do not need if it succeeds or not, so even if you are then able to reply "something" to the browser, how will you handle failures?
The generic solution to cases like this is:
The action triggered by the HTTP visit just records the request, like in a DB or something
Separately a process polls the DB for tasks to do and executes them; this is completely separate from any webserver; when job has completed (success or failure), the DB will be updated in the same way
The process at 1) can regularly poll the DB to see the status (pending or completed or failed) and show user a message, like during its visit, with some kind of auto-refresh, and/or provide a specific separate page where the user would be able to track status of its operation
Should I use an ACME PHP library recommended by LetsEncrypt - https://letsencrypt.org/docs/client-options/
You can use any library in any language that properly implements the ACME protocol.
Let's Encrypt only recommends one software: certbot. All the rest listed in that page are examples of client libraries/programs that are expected to work.
Do I need to create virtual hosts for each domain pointing to www.mywebapp.example?
Yes, specially if they are using each of them a specific certificate, otherwise the webserver will not be able to identify the proper certificate to return at the beginning of the TLS handshake based on which hostname the browser gave (inside the SNI extension used at beginning of TLS handshake)
Or you can use some Apache features of mass virtual hosting, such as https://httpd.apache.org/docs/2.4/mod/mod_vhost_alias.html
As is however this may probably mean a single certificate with all names added to it, which can technically work until some amount of names, but can create non technical problems (like seeing all the names, etc.)
Of course other software, like Nging/HAProxy can provide more advanced features, where you do not need to configure things virtual host per virtual host, even with different certificates, you just put things in place with specific naming and the webserver will match things as needed.
Do I need to edit /etc/httpd/conf.d/ssl.conf to add the new virtual hosts?
Yes, or any other file as long as you use Include or similar.
If you manage many different and separate websites, it may be simpler to have one configuration file per website, including its certificate paths and so on, many Linux distributions install Apache in such a way that you have /etc/httpd/sites-enabled/ for that feature. If not, you can do the same yourself.
Can multiple domains use the same DocumentRoot path?
Of course yes.
PS: please stop saying SSL, the protocol is named TLS, invented 20 years ago. SSL is long gone, current recommandations are to run TLS 1.2 except if good reasons to also allow 1.1 and 1.0 which have vulnerabilities. There are no "SSL certificate" either for this reason and because they are a misnommer. TLS can work without certificates and these certificates can be used outside of TLS, like in S/MIME. They are X.509 certificates.
We have develop a CURL function on our application. This curl function is mainly to map the data over from 1 site to our form-field in our application.
However, this function has been working fine all the while and ready for use for more than 2 months. Yesterday, this fucntion was broken down. the data from this website is no longer able to map over. We are trying to find out why the problem is. When we troubleshooting, it shows that there is response timeout issue.
To re-ensure there were nothing wrong on our coding and our server performance is working, we have duplicates this instance to another server and try out the function. It was working perfectly.
Wondering if any one out there facing such problem?
What could the possibility to cause this issue?
When we are using cURL, will the site owner know that we are calling their data to map into ours server application? If so, is there a way that we can overcome this?
Could be the owner that block our server ip address? tht's why it function works well on my another server but not in the original server?
Appreciate your help on this.
Thank you,
Your problem description is far too generic to determine a specific cause. Most likely however there is a specific block in place.
For example a firewall rule on the other end, or on your end, would cause all traffic to be dropped, thus causing the timeout. There could also be a regular network outage between both servers, but that's unlikely.
Yes, they will see it in their Apache (or IIS) logs regularly. No, you cannot hide from the server logs - it logs all successful requests. You either get the data, or you stay stealthy. Not both.
Yes, the webserver logs will contain the IP doing all the requests. Adding a DROP rule to the firewall is then a trivial task.
I have applied such a firewall rule to bandwidth and/or data leechers a lot of times in the past few years, although usually I prefer the more resilient deny from 1.2.3.4 approach in Apache vhost/htaccess. Usually, if you use someone else's facilities, it's nice to ask for proper permission - lessens the chance you get blocked this way.
I faced a similar problem some time ago
My server IP was blocked from the website owner
It can be seen in the server logs. Google Analytics, however, won't see this, as cURL doesn't execute javascript.
Try to ping the destination server from the one executing the cURL.
Some advices are:
Use a browser header to mask your request.
If you insist on using this server, you can run trough a proxy.
Put some sleep() between the requests.
I have a really weird behavior going on.
I'm hosting a tracking software for users, that mainly logs mobile traffic. Now, the path is as follows:
1. My client gets a php code snippet to put in his website.
2. This code sends a cURL post (based on predefined post fields like: visiotr IP, useragent, host etc) to my server.
3. my server logs the data, and decide what the risk level is.
4. it then responds the client server about the status. That is, it sends "true" or "false" back to the client server.
5. client server gets that r
esponse, and decides what to do (load diffrent HTML content, redirect, block the visitor etc).
The problem I'm facing is, for some reason, all the requests made from my client's server to my server, are recorded and stored in the a log file, but my clients report of click loss as if my server sends back the response, but their server fails to receive those responses or something.
I may note that, there are tons of requests every minute from different clients' servers, and from each client himself.
Could the reason be related to the CURL_RETURNTRANSFER not getting any response ? or, maybe the problem is cURL overload ?
I really have no idea. My server is pretty fast, and uses only 10% of its sources.
Thanks in advance for your thoughts.
You touched very problematic domain - high load servers, you problem can be in so many places, so you will have to really spend time to fix it, or at least partially fix.
First of all, you should understand what is really going on, check out this simplified scheme:
Client's php code tries to open connection to your server, to do this it sends some data via network to your server
Your server (I suppose apache) tries to accept it, if it has resources - check max connections properties in apache config
If server can accept connection it tries to create new thread (or use one from thread pool)
After thread is started, it runs your php script
Your php script do some work, connecto to db and sends response back via network
Client waits till the answer from p5 or closes connection because of timeout
So, at each point you can have bottleneck:
Network bandwidth
Max opened connections
Thread pool size
Script execution time
Max database connections, table locks, io wait times
Clients timeouts
And it is not a full list of possible places where problem can occur and finally lead to empty curl response.
From the very start I suggest you to add logging to both PHP codes (clients and servers) and store all curl_error problems in some text file, at least you will see what problems occur often.
I'm curious to know how to stop Apache from logging every URL I search with CURL.
My PHP script opens a few hundred thousand URLs, scans them, takes a tiny bit of info, closes, and then opens the next.
I discovered after opening the access log that each and every URL opened with CURL is written to the access log.
::1 - - [01/Dec/2010:18:37:37 -0600] "GET /test.php HTTP/1.1" 200 8469 "-"..."
My access log is almost 45MBytes large. Help anyone?
This is the purpose for access log - recording any incoming traffic
In order to effectively manage a web server, it is necessary to get feedback about the activity and performance of the server as well as any problems that may be occurring. The Apache HTTP Server provides very comprehensive and flexible logging capabilities. This document describes how to configure its logging capabilities, and how to understand what the logs contain.
source: http://httpd.apache.org/docs/trunk/logs.html
Of course, you have the option to disable logging (preferable not)
If all of your curl requests are coming from a single or otherwise manageable group of IPs you can exclude them from your logs with a configuration similar to the following:
# Set your address here, you can do this for multiple addresses
SetEnvIf Remote_Addr "1\.1\.1\.1" mycurlrequest
CustomLog logs/access_log common env=!mycurlrequest
You can do something similar with the user agent field which by default will indicate that it's curl.
You can read more here:
http://httpd.apache.org/docs/2.2/logs.html#accesslog (conditional logging is the last section under this header)
and here
http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html#setenvif
If you want to conditionally exclude logging I would to it by the most precise method possible such as the ip address. In the event the server is externally accessible you probably don't want to find yourself NOT logging external requests from curl.
Using conditional logging you can also segment your logging if you want to multiple files one of which you could roll more frequently. The benefit of that is you can save space and at the same time have log data to help research and debug.
See the Apache manual, about Conditional Logs. That may be what you are looking for.
I have a script that checks responses from HTTP servers using the PEAR HTTP classes. However, I've recently found that the script fails on FTP servers (and probably anything that's not HTTP or HTTPS). I tried Google, but didn't see any scripts or code that returned the server status code from servers other than HTTP servers.
How can I find out the status of a newsgroup or FTP server using PHP?
EDIT: I should clarify that I am interested only in the ability to read from an FTP server and the directory that I specify. I need to know if the server is dead/gone, I'm not authorized to read, etc.
Please note that, although most of the time I'm language agnostic, the entire website is PHP-driven, so a PHP solution would be the best for easy of maintainability and extensibility in the future.
HTTP works slightly differently than FTP though unfortunately. Although both may look the same in your browser, HTTP works off the basis of URI (i.e. to access resource A, you have an identifier which tells you how to access that).
FTP is very old school server driven. Even anonymous FTP is a bit of a hack, since you still supply a username and password, it's just defined as "anonymous" and your email address.
Checking if an FTP server is up means checking
That you can connect to the FTP server
if (!($ftpfd = ftp_connect($hostname))) { ... }
That you can login to the server:
if (!ftp_login($ftpfd, $username, $password)) { ... }
Then, if there are further underlying resources that you need to access to test whether a particular site is up, then use an appropiate operation on them. e.g. on a file, maybe use ftp_mdtm() to get the last modified time or on a directory, see if ftp_nlist() works.
Wouldn't it be simpler to use the built-in PHP FTP* functionality than trying to roll your own? If the URI is coming from a source outside your control, you would need to check the protocal definition (http:// or ftp://, etc) in order to determine which functionality to use, but that is fairly trivial. If there is now protocal specified (there really should be!) then you could try to default to http.
see here
If you want to read specific responses you will have to open a socket and read/write data manually.
<?php
$sock = fsockopen($hostname, $port);
?>
Then you'll have to fput/fread data back and forth.
This will require you to read up on the FTP protocol.