I am looking into implementing a virus scanner into a web application I am creating that allows the user to upload files. All the background functionality is completed, however I am wary that a malicious user may upload a virus and other users download that file.
I have been researching for a few months now on how to implement a scanner into my web application.
I have located various online virus scanners such as MetaScan-Online and VirusTotal.
I have read into the documentation that they have provided, however I am still confused and am unsure if I can implement these applications into my applications using the API.
Can I?
And if so, is there another virus scanner that enables a whole folder of files to be scanned simultaneously?
If the anti-virus force is strong in you, then you can probably implement a service class and upload the incoming files to one of the public scan services.
Keep in mind that they are limiting the accepted file size and number of files and that they don't store the scan reports forever.
MetaScan
The public API for Metascan is described here: https://www.metascan-online.com/public-api#!/
There is also a PHP checker available. But it uses v1 of their API and looks outdated. Maybe contact them to get an update version using API v2.
https://www.metascan-online.com/apps#!/php-metascan-online-checker
VirusTotal
The public API is described here: https://www.virustotal.com/de/documentation/public-api/
There are multiple libraries for PHP available, just to mention one of them
https://github.com/jayzeng/virustotal_apiwrapper
Local clamAV scan after upload
Another solution is to simply trigger a clamav scan by using clamscan after a file was uploaded to your server. That means upload to sandboxed av-scan folder, scan, drop (if bad), or keep (if ok) and finally move to upload folder. This is slow, because signatures have to be loaded each time the clamscan command is invoked from PHP.
With sandbox folder i mean an restricted system environment for controlling the resources better, e.g. an "upload directory" with restricted or removed read/write/exec permissions (user/group/other), isolated from the application, not accessible from the web and with restricted PHP capabilities (think of disable_functions directive).
When your server runs a clamav daemon (clamd) its possible to invoke clamdscan on the folder. This solution is fast, because the signatures a kept in memory.
You'll have to handle the sending of folder yourself, they'll not bez able (as an external service) to get the list of file for a defined folder.
VirusTotal is providing a public API to scan your file, it's a good start. You could implement multi threading and store the result for each file. This way you'll avoid sending multiple time the same file.
Related
I have not been able to find solid information on preferred (best practices) and/or secure methods to allow php to access config or other types of files on a linux server not contained in the public web directory or owned by the apache user so I'm hoping to find some answers here.
I am a fairly competent PHP programmer but am increasingly tasked with writing web applications (most of which are not publicly accessible via the web however) that require updating, changing or adding to config files or files generated by some service or application on the server.
For instance, I need to create a web interface that will view, add or remove entries from a /etc/mail/spamassassin/white-list.cf file owned by root.
Another scenario is that I need php to parse mime messages in /var/vmail that are owned by user vmail.
These are just a couple examples, there will be other files in locations owned by other processes/users. How can I write PHP applications that securely access and manipulate these files without opening security risks?
If I were needing to implement something like this, I would probably look at using something like sudo to fine-tune permissions. I'm not a Linux CLI expert, so I'm sure there are issues that I haven't taken into account when typing this out.
I would probably determine what tasks need to be done, and would write a separate script for each task that needs to be completed. Using sudo, I'd assign the necessary level of permissions for that script only.
Obviously, as the number of tasks increase, so would the complexity and the amount of work involved. I'm not sure how this would affect you at the moment.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Best way to monitor file system changes in linux
I need your help. How can I log activities done using SSH on a linux server - activities like create file or dir, delete file or dir, rename file or dir for a particular path. I need some solution any bash, python or php script or if there is any option in linux using which I can watch all activities done on a particular path or folder. I need to use those logs for syncing purpose.
OK, Let me explain you the entire scenario. I am working on sync tool we are using Samba for sharing all the files and folders and I need these files to be synced across the network. I grep samba log to watch the activities done by clients like create file or folder, delete file and folder and rename file or folder I am using these log for my syncing tool and its working fine. But I am only getting logs when changes are done using samba - if the change are done using SSH those activities are not logged and will not be synced. So I need to grep log for the changes made using SSH for a particular path (for example: /mnt/test) - changes made in test folder like create delete and rename.
As I understand what's happening here is this.
There is a Samba server that exports a filesystem with multiple users and a bunch of users that have direct access to the filesystem (they log on via SSH). And this filesystem needs to be replicated to another location.
The TS is developing a tool to perform the replication.
There are at least two options here.
A more conventional way to do this would be to run rsync between the two locations at regular intervals. Thus the replicas will not be always consistent, but it is easy and the system is partition tolerant and available. That is this this method chooses "A" and P" from the "CAP" theorem.
Another method inspired by the popularity of Dropbox-like cloud storage and instant replication is to watch the filesystem. That can be accomplished with inotify or fam.
The interfaces for inotify are available in most scripting languages including Perl, Python and PHP. This trades consistency for availability. That is until a large file has been replicated it would not be accessible on the other side.
The interfaces for FAM are available in PHP and probably other languages. See the linked question for a discussion of different filesystem monitoring APIs.
The first option is essentialy a one-liner. The second option should not be too hard either (look at the Dropbox daemon sources for an example).
Note: Replication is a recurring topic at Serverfault.com.
I am designing a web-based file-managment system that can be conceptualised as 3 different servers:
The server that hosts the system interface (built in PHP) where users 'upload' and manage files (no actual files are stored here, it's all meta).
A separate staging server where files are placed to be worked on.
A file-store where the files are stored when they are not being worked on.
All 3 servers will be *nix-based on the same internal network. Users, based in Windows, will use a web interface to create an initial entry for a file on Server 1. This file will be 'uploaded' to Server 3 either from the user's local drive (if the file doesn't currently exist anywhere on the network) or another network drive on the internal network.
My question relates to the best programmatic approach to achieve what I want to do, namely:
When a user uploads a file (selecting the source via a web form) from the network, the file is transferred to Server 3 as an inter-network transfer, rather than passing through the user (which I believe is what would happen if it was sent as a standard HTTP form upload). I know I could set up FTP servers on each machine and attempt to FXP files between locations, but is this preferable to PHP executing a command on Server 1 (which will have global network access), to perform a cross-network transfer that way?
The second problem is that these are very large files we're talking about, at least a gigabyte or two each, and so transfers will not be instant. I need some method of polling the status of the transfer, and returning this to the web interface so that the user knows what is going on.
Alternatively this upload could be left to run asyncrhonously to the user's current view, but I would still need a method to check the status of the transfer to ensure it completes.
So, if using an FXP solution, how could polling be achieved? If using a file move/copy command from the shell, is any form of polling possible? PHP/JQuery solutions would be very acceptable.
My final part to this question relates to windows network drive mapping. A user may map a drive (and select a file from), an arbitrarily specified mapped drive. Their G:\ may relate to \server4\some\location\therein, but presumably any drive path given to the server via a web form will only send the G:\ file path. Is there a way to determine the 'real path' of mapped network drives?
Any solution would be used to stage files from Server 3 to Server 2 when the files are being worked on - the emphasis being on these giant files not having to pass through the user's local machine first.
Please let me know if you have comments and I will try to make this question more coherant if it is unclear.
As far as I’m aware (and I could be wrong) there is no standard way to determine the UNC path of a mapped drive from a browser.
The only way to do this would be to have some kind of control within the web page. Could be ActiveX or maybe flash. I’ve seen ActiveX doing this, but not flash.
In the past when designing web based systems that need to know the UNC path of a user’s mapped drive I’ve had to have a translation of drive to UNC path stored server side. I did have a luxury though of knowing which drive would map to what UNC path. If the user can set arbitrary paths then this obviously won’t work.
Ok, as I’m procrastinating and avoiding real work I’ve given this some thought.
I’ll preface this by saying that I’m in no way a Linux expert and the system I’m about to describe has just been thought up off the top of my head and is not something you’d want to put into any kind of production. However, it might help you down the right path.
So, you have 3 servers, the Interface Server (LAMP stack I’m assuming?) your Staging Server and your File Store Server. You will also have Client Machines and Network Shares. For the purpose of this design your Network Shares are hosted on nix boxes that your File Store can scp from.
You’d create your frontend website that tracks and stores information about files etc. This will also hold the details about which files are being copied, which are in Staging and so on.
You’ll also need some kind of Service running on the File Store Server. I’ll call this the File Copy Service. This will be responsible for coping the files from your servers hosting the network shares.
Now, you’ve still got an issue with how you figure out what path the users file is actually on. If you can stop users from mapping their own drives and force them to use consistent drive letters then you could keep a translation of drive letter to UNC path on the server. If you can’t, well I’ll let you figure that out. If you’re in a windows domain you can force the drive mappings using Group Policies.
Anyway, the process for the system would work something like this.
User goes to system and selects a file
The Interface server take the file path and calls the File Copy Service on the File Store Server
The File Copy Service connects to the server that hosts the file and initiates the copy. If they’re all nix boxes you could easily use something like SCP. Now, I haven’t actually looked up how to do it but I’d be very surprised if you can’t get a running total of percentage complete from SCP as it’s copying. With this running total the File Copy Service will be updating the database on the Interface Server with how the copy is doing so the user can see this from the Interface Server.
The File Copy Service can also be used to move files from the File Store to the staging server.
As i said very roughly thought out. The above would work, but it all depends a lot on how your systems are set up etc.
Having said all that though, there must be software that would do this out there. Have you looked?
If iam right is this archtecture:
Entlarge image
1.)
First lets sove the issue of "inter server transfer"
I would solve this issue by mount the FileSystem from Server 2 and 3 to Server 1 by NFS.
https://help.ubuntu.com/8.04/serverguide/network-file-system.html
So PHP can direct store files on file system and dont need to know on which server the files realy is.
/etc/exports
of Server 2 + 3
/directory/with/files 192.168.IPofServer.1 (rw,sync)
exportfs -ra
/etc/fstab
of Server 1
192.168.IPofServer.2:/var/lib/data/server2/ /directory/with/files nfs rsize=8192,wsize=8192,timeo=14,intr
192.168.IPofServer.3:/var/lib/data/server3/ /directory/with/files nfs rsize=8192,wsize=8192,timeo=14,intr
mount -a
2.)
Get upload progress for realy large files,
here are some possibilitys to have a progress bar for http uploads.
But for a resume function you would have to use a flash plugin.
http://fineuploader.com/#demo
https://github.com/valums/file-uploader
or you can build it by your selfe using the apc extension
http://www.amwsites.com/blog/2011/01/use-a-combination-of-jquery-php-apc-uploadprogress-to-show-progress-bar-during-an-upload/
3.)
Lets Server load files from Network drive.
This i would try with a java applet to figurre out the real network path and send this to server, so the server can fetch the file in background.
But i never didt thinks like this before and have no further informations.
I am planing to build a CMS-system.
The CMS system will be located and fully administrated at:
www.mycompany.com/customer
The final website will be located at:
www.customer.com
I need to figure out the best way to copy files (eg. images)
from: www.mycompany.com/customer/media
to: www.customer.com/media
Note: The CMS and customer page will be located on different hosts. And I want to build this function using PHP.
Some thoughts:
The optimal solution would be if the two directories could be cloned automaticly, no matter how the images are uploaded or updated. Maby if there is a way to detect changes to www.mycompany.com/customer/media, then www.customer.com/media could be notifyed about it and send a request to update the image.
A wish would also be that images only could be accessed from www.mycompany.com/customer/media if logged in to the CMS :S
Any tips?
You should not use FTP (insecure), or PHP for the replication,
try rsync instead :-
What is rsync ?
http://en.wikipedia.org/wiki/Rsync
rsync is a software application and network protocol for Unix-like and Windows systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression and recursion.
In another word, is designed meant for mirroring or replicating (is industry standard)
In general,
setup public key to allow source server to able to ssh into destination server
setup a cronjob in the source server to do rsync
What does the cronjob do ?
In nutshell, it should rsync the selected source directory to destination server,
a quick example :-
* * * * * rsync -avz /home/www.mycompany.com/www $HOST:/home/www.customer.com/www
^ source server directory ^ destination server,
and directory
However, rsync is too hard to describe in few sentences, you can take a look :-
http://www.cyberciti.biz/tips/linux-use-rsync-transfer-mirror-files-directories.html (as a start)
Other possibilities is make use of version controlling software, like -:
git
svn
Or make use on CDN (like #Amir Raminfar has mentioned), which itself is already a complete solution for file distribution.
Just make an ftp call or scp call from mycompany.com to customer.com and upload the file.
Alternately, you can use a CDN which is probably a little too much work but it will give you optimal solution and with great performance. You can upload to a common server and have customer.com pull from that website for the images. This gives you the benefit for parallel download for the assets.
Finally, you can run some kind of web service on customer.com that can accept uploads.
I wouldn't do what you suggested which is notification system because it won't be exactly instantaneous and it will result in some time before actually seeing the image.
You can create DB for www.mycompany.com. But host must allows remote access to DB.
Create crontab for customer.com which will be check new records within remote DB. This script could be allows for registred users.
You should create htaccess for www.mycompany.com images folder. To allow access from itself and customer.com to download (say copy) images.
In this way all registred users can download any image. It could be wrong for you.
If you need to limit access for each image then you should to store images in DB and generate encrypt url for each image and request. In this case you shold to use remote DB or SOAP service for getting images.
I'm building a web server out of a spare computer in my house (with Ubuntu Server 11.04), with the goal of using it as a file sharing drive that can also be accessed over the internet. Obviously, I don't want just anyone being able to download some of these files, especially since some would be in the 250-750MB range (video files, archives, etc.). So I'd be implementing a user login system with PHP and MySQL.
I've done some research on here and other sites and I understand that a good method would be to store these files outside the public directory (e.g. /var/private vs. /var/www). Then, when the file is requested by a logged in user, the appropriate headers are given (likely application/octet-stream for automatic downloading), the buffer flushed, and the file is loaded via readfile.
However, while I imagine this would be a piece of cake for smaller files like documents, images, and music files, would this be feasible for the larger files I mentioned?
If there's an alternate method I missed, I'm all ears. I tried setting a folders permissions to 750 and similar, but I could still view the file through normal HTTP in my browser, as if I was considered part of the group (and when I set the permissions so I can't access the file, neither can PHP).
Crap, while I'm at it, any tips for allowing people to upload large files via PHP? Or would that have to be don via FTP?
You want the X-Sendfile header. It will instruct your web server to serve up a specific file from your file system.
Read about it here: Using X-Sendfile with Apache/PHP
That could indeed become an issue with large files.
Isn't it possible to just use FTP for this?
HTTP isn't really meant for large files but FTP is.
The soluton you mentioned is the best possible when the account system is handled via PHP and MySQL. If you want to keep it away from PHP and let the server do the job, you can protect the directory by password via .htaccess file. This way the files won't go through the PHP, but honestly there's nothing you should be worried about. I recommend you to go with your method.