I have a file from a library with size > 50MB so I cannot deploy it with Git in my instances. I include this file in some of my PHP scripts, so what should I do in order to leave my instance replicable and include this file in my scripts?
I can store it in a S3 bucket but im not sure if that's a good practice (including external files).
Files that are needed but not practical to keep in a repository are perfect for S3. I typically create a [companyname]-ops or [companyname]-assets S3 bucket with narrow access- such as a read-only IAM role for standard machines.
Part of the deployment process is to push (or pull) your code, and to pull assets from S3.
Obviously this can be done a million ways. I tend to think of code repos, databases, and s3 as tools that have their own uses for deployments.
Storing the files on S3 is one option that I have seen quite often.
Alternatively you can bake your own AMI with the file already included, so you dont require extra bootstraping. This should also speed up the whole replication process.
I have a set of confidential files in Google Cloud Storage that I allow people to download one at a time using CloudStorageTools::serve. It would be more convenient if they could select a group of files and download them in a zip file. I am using PHP with Google App Engine and see that PHP has a ZipArchive class.
I don't see a straightforward way to create a zip file using CloudStorageTools:serve, which for times sake would be my preferred method. If someone knows how to do this, that would be very helpful.
If it's not possible, it would seem a lot of other options may work, from momentarily turning the files into public accessible, which would give them a public URL that I could use to create the zip files. I would prefer to avoid this because of the confidentiality issue. Another option would be to temporarily copy them to a folder in Google App Engine, zip them and then delete them. Again, I have concerns with confidentiality. Both these methods seem somewhat time consuming to setup. So if someone has a faster/better method, I would really appreciate the help. Thanks!
I'm building website with billions of images. I'm little confused to store images in single directory. how many images can be stored in single directory. or it slow down the server ?
Have you considered object storage such as AWS S3? http://aws.amazon.com/s3/
As for performance, I think it depends on the file system you intend to use. Some file systems index directory content in a linear manner, others use more efficient algorithms. It also depends whether any of your system services will need to scan the directory regularly.
I found this: http://events.linuxfoundation.org/slides/2010/linuxcon2010_wheeler.pdf
In this question: https://serverfault.com/questions/43133/filesystem-large-number-of-files-in-a-single-directory
I knew that PHP is able to read file content by different ways, for example: fread, file_get_contents, file, readfile, etc.
Currently, I am looking for an API that can read real index of files and folders in specific partition or folder, for example:
drive d:\ in windows contains three folders (folder1, folder2, folder3), and each folder contains some files, we can get these directory structure using PHP (opendir, scandir, readdir, etc) and list them as I want, however, windows saved file and folder names inside hard-disk with their attributes (size, last modified, created on, etc).
How I can read hard-disc using PHP and retrieving all file and folder attributes for a specific path?
for instance, if we consider last modified time we can use (filemtime()) function, but this attribute not saved inside the file, its saved some where else inside hard-drive, other attributes also saved in other location not inside the file.
When windows user copying file from flash-drive to local hard, windows will copy all file and folder attributes and saves them inside local hard drive. When using PHP for copying file, it depends on OS to handle this job, its not native support (as I think) for file and folder operations.
Do you have any idea?
There are many recovery program that uses this technology for reading hard-drive indexes, however, for PHP: I cant find any source for this problem.
Applications if I get correct answer:
I can check if such file securely deleted from my hard-drive? I can create secure delete application using PHP, or clearing hard-drive indexes for a given file.
Your help appreciated.
Problems with the proposition
The attributes of files, such as timestamps, permission flags etc, are stored in the file system (FAT, NTFS, Ext3 etc). As you say some of them can be read using PHPs different file and directory methods, but they all act through the OS file system abstraction and cant have access to block level information on the disk, such as what precise byte on disk stores the archive flag for file X. The whole point of the OS and FS is to abstract away this information from the user/client programs.
As suggested there are external tools, written in c or similar, that does have this access and that you can call from inside PHP. If you want a 'native' PHP way of doing this you'll have to compile a c extension for PHP that exposes these low level functions to you.
I'd say external tools is the way to go if you want to stick with PHP but for the task at hand, as far as we can see from your description, I'd go with another language that has more low level access. Like C or C++. PHP is a high level language for HTML pre processing and as such is a poor choice for low level system programming.
Practical advice
After looking through the PHP documentation and assorted third party libraries:
An of the shelf solution for reading file system information on a file allocation table level doesn't exist for PHP. The lowest level you get is the fstat() function, and that is not very far for what you want.
External tools
No mater exactly what you want to do there is probably a small binary that does it. PHP can be integrated with these programs, as suggested elsewhere, via the exec() function. This is probably the easiest approach for you unless you have serious amounts of time and/or development resources to devote to this problem.
Wrapping a library
There are libraries that solves this problem for you, written in low level languages. An open source library can be wrapped with SWIG to expose it to PHP. This will give you access to the low level methods you need, but it's a non trivial task. These kind of libraries also often require sole access to the device while they work on it, something that is difficult to achieve in most normal operating environments.
Note also that you will probably need a library per file system. Microsofts VFAT extension to FAT12/16/32 requiers a licens to use. So if you want to work with FAT and have files with long names (not 8.3 format) you'll have to fork up some dough to be legit.
Low level implementation
A last middle ground would be to write your own CLI tool that uses an external library to access the low level FS functions. You can then use exec() from inside PHP to interact with your own implementation.
This might be a reasonable path if you cant find an existing tool that solves your problem and you are not willing to spend the time to wrap a library.
In closing
You give a very narrow problem description with little to go on as for what the application is about. A broader discussion (in another forum) might yield better results since the problem might be better solved in another way entirely.
I found something on PHP.net which appears to do what you want:
http://php.net/manual/en/function.readdir.php#103418
Edit: I mis-understood the question. Attributes such as the last modified time, last accessed date and the like are stored in the file systems master file table. As far as I can tell, this isn't accessible with PHP, and if you were to write your own method to do this then you'd also have to account for different file systems as they all handle the storage of these attributes in their own unique way.
It could be that to get all of the information you're looking for is not possible with PHP without writing some form of extension to PHP itself.
Edit 2: Upon researching a little more...
http://php.net/manual/en/function.fileinode.php
This function could be an interesting one to look at.
Well if I understand correctly you just want to securely delete a file. You can just call [shred][1]
[1]: http://linux.die.net/man/1/shred via system or exec if you are on linux and you are good to go
Im planning to add file manager (very basic once) because I never used FTP functions, and it looks easier (FTP connection loses when scripts is done). I would simply use POST request (or what should I?) instead of FTP functions. Is it good idea? Anyone knows restrictions?
As far as I can see only FTP functions are to post and receive files.
What you need to do is add dynamic form where you can select multiple files and upload them to specific directory of your chose.
You will need to get all available directories and files in them, probably with some kind of recursive function. More optimal way is to get directories/files of current folder and when you click on folder it will get files/folder for it.
Can it be done - sure. Is it a good idea - no. People will have access for uploading malicious files, we are not talking about images here, php scripts, shell scripts, executable viruses and so on...
If you are doing this only for yourself, for file posting and receiving I suggest you to use FTP clients for that.
I wouldn't recommend it, but it's probably best to use a 3rd party tool, rather than to write your own.
PHP File Manager
PHPfileNavigator2
FileManager
...
Keep in mind that both PHP and your webserver can put certain restrictions on the size of files that you can transfer, it is of course possible to change these in the configuration files.