The use case:
8 servers of 300 php-fpm concurrent child process each, produces records to Apache Kafka.
Each one produces 1 Kafka record, 1000 records per second.
Why do we need so many connections?
We have a web API, that is getting 60K calls per minute. Those requests are doing many things and processed via thousands of web php-fpm workers (unfortunately). As part of the request handling, we produce events to Kafka.
The problem:
I cannot find a way to persist connections between php-fpm web requests, creating some that seems to me so inefficient as might hit Kafka bounderies(will it?).
The result is 1000 producer connections per second getting established, sending one single record ach and getting closed just after.
I read here https://www.reddit.com/r/PHP/comments/648zrk/kafka_php_71_library/ that php-rdkafka is efficient, but I dont know if it can solve this issue.
I thought that Opcache might be handy to reuse connections but I cannot find a way to do it.
The question
Is establishing and closing 1000 connections per second fast and cheap or is a proxy with cheap connectivity that will reuse Kafka connection a must for such a use case?
Related
ReactPHP http server for each user, Is this a good idea?
In my application:
Each logged on user sends and receives data from server. In average one request per second.
After server response, the server have some extra work to do, which is related to specific user.
I can simply build new ReactPHP http server for each user who logs, and release the server after the user log out.
Is this will work? Am i missing something ?
No, it's not a good idea. You need a separate port per user in that case to route the user to the right server. That'd quickly exhaust your ports.
If you have blocking tasks within the event loop and want to use multiple processes because of that, just stick to traditional PHP with mod_php or php-fpm and start a new event loop for each process, do your thing and then exit.
If you don't have any blocking operations and everything is non-blocking, you can just use a single server and it handles all the things.
I'm not sure if exhausting ports would be the issue. Other services that do just this such as WebRTC SFUs. With 65,535 ports available that your talking 30,000+ concurrent TCP connections.
However, with that many users first obvious problem would be memory. At 10 mb just to start up PHP, that would be 300+ gb of memory without including a single line of code or actually doing anything. If your working with a seriously trimmed php binary you can get down to 4 or 5 mb, so at 5,000 concurrent users you would have around 25 gb.
But the real problem is that it would result in thousands of processes, which is impossible to work around. This would be entirely wasteful considering ReactPHP's eventloop can handle 10k users within a single process. Not saying a single PHP process can do the work for that many users (except maybe the most basic chat) but ReactPHP can handle the IO. Throwing them all into their own process though would a nightmare.
The basic idea has been tried in other languages by giving each user their own thread, but even in C/C++ this is quickly proven to be a bad design.
I need to write a socket server that will be handling at least something about 1000 (much more in future) low-traffic permanent connections. I have made a draft version on PHP for testing purposes (we are developing a monitoring hardware, so we needed to develop and test a conversation protocol and hardware capabilities), which suited me very well when i had just a couple of clients connected. But when the amount of connections had grown to ten, some critical issues appeared. Here some info about server architecture:
I have a master process, which waits for socket connections and on connecting creates a child process (that serves this connection from now on) using pcntl_fork(). Also i am setting up a PDO connection to MySQL in master process. All the child processes are sharing the same single PDO object. At first i was afraid of getting some collisions during simultaneous queries, but i haven't encountered them, even through stress-test (10 children were making queries in the loop without stopping). But there is usleep(500000) in each child, so it could be luck, though i had this testing running for a couple of hours. But such load should not be present even at 1k clients connected, due to rare conversations between them and server.
So here is my first question: is it safe to use single PDO object for a big amount of child processes (ideally there would be around 1000)? I can use single connection for each child, but MySQL doesn't support nearly as much connections.
The second issue is in getting parasite MySQL connections. As i mentioned before, i have only one PDO object. But when i have more than one clients connected, and after they had run some queries, i see in mytop that there is more than one DB connection, and i could not find any correlation between the amount of connections and amount of child processes i have. For example i have 3 childs, and 5 DB connections. I tried to establish persistent connections, and it didn't changed anything.
Second question: Is it PDO who makes those additional connecitons to MySQL, or it is the MySQL driver? And is there a way to force them to use one connection? I don't think it could be my fault, my code prints an alert to console every time i call method which creates PDO object, and that happens only once, at the script start, before forking. After that i only run querys out of children, using parent's PDO object. Once again i can not afford to have so many connections due to MySQL limitations.
Third question: Will be one thousand of socket connections a problem by itself? Aside of the CPU and database load, i mean. Or i should do some amount of lesser servers (128 connections for example), that will tell the clients to connect to other one if max number of connects is exceeded?
Thanks in advance for your time and possible answers.
Currently your primary concern should be your socket server architecture. Forking a process for each client is super heavy. AFAIK an average PC can tolerate around 2000 threads and it's not going to work fast. Switching between processes means that CPU should save its state in memory, and if you have enormous amount of processes, CPU will be busy with memory IO and will have little time for actually doing stuff.
You may want to look at Apache for inspiration. In Apache they use a fixed amount of worker processes/threads, each process/thread working with multiple clients via select function and sockets in a non-blocking mode. This is a far more robust approach.
Regarding database IO, I would spawn a process/thread that would be the sole owner of database connections. Worker processes would communicate with the DB IO process using IPC (in case of processes) or lock-free queues (in case of threads). This approach makes you independent of PDO implementation details (if it is thread safe or does it spawn connections etc).
P. S. I suspect that you actually spawn new PDO objects with forking (forking merely means making a copy of a process with its memory and everything inside it) and PDO objects create and shut down connections on demand. It may explain why you're not seeing correlation between low traffic clients and DB connections.
The scenario is - I am building a message queue model using RabbitMQ and phpamqplib. This model will have 15 programs each program will consume a message from a queue and publish a message to another queue. All these queues are different (i.e. around 30 queues). But I want to use only 2 connections across all these programs one for publishing and another one for consuming. I don't want to create broker connections in each of the program. I am not able to understand how to do it? Any help? Thanks in advance.
If you want to use 2 connections then 15 producers and consumers should be part of a single process and run as threads. In addition two threads one for consuming and other for publishing.
The consumer thread consumes messages and pushes them to remaining worker thread pool.
Once the worker threads have completed their work, response is pushed to an internal storage inside publisher,which in turn pops the response onto rabbit queues.
. Few points to keep in mind are:
Throughput: Number of consumers and producers is decided on basis of throughput you want to achieve for your application.
Scalability, if you have fixed number of consumers and producers then you might be able to scale your application to a limit.
Flow control: number of consumers can be crucial in avoiding connection based flow control.
Internal message caching by consumer thread (Qos). Set a well defined QOs value as per the throughout desired.
Also explore if multi-threading is supported by amqp library you desire to use. If yes then you could share the connection across threads.
I wanted to know if I forked a processes after I bonded the IP and port of the server. Will the fork be able to accept the connection. To extend on that if I have 10 forks running all trying to accept is there a chance that more then 1 could accept the same connection or is there some locking on that?
A few days ago I felt like writing writing a http server in php. So it can handle more then one connection at a time. The master processes accepted the connection reads the data in and passes it to a thread via a unix socket. So far on my laptop i can get 1000 connections a second on a little page that gives the current date and time. One of the bottle necks is the master processes. Originally i would have loved to gotten the file descriptors of the connections and passed those to the sockets and have them read the data in then processes it.
Yes, the forked children will be able to accept new connections on the same (inherited) listening socket.
Assuming you're using a blocking socket_accept() in all your child processes, you shouldn't experience any performance issues even if it ramps up to 100 processes; the operating system will wake up one child process to handle the connection.
It should be mentioned that it's good practice to benchmark it, using ab or similar load generator tools.
I'm using redis (2.6.8) with php-fpm and phpredis driver and have some trouble with redis latency issues. Under certain load first request to redis from our application takes about 1-1.5s and redis-cli --latency shows the same latency.
I've already checked the latency guide.
We use redis on the same host with Unix sockets
slowlog has no entries longer 5ms
we don't use AOF
redis takes about 3.5Gb memory of 16Gb available (i suppose it's not too much)
our system is not swapping
there is no other process doing disk I/O
I'm using persistent connections and amount of connected client is varying from 5 to 25 (sometimes strikes to 60-80).
Here is the graph.
It looks like problems starts when there are 20 or more simultaneously connected clients.
Can you help me to figure out where is the problem?
Update
I investigated the problem and it seemed like redis did not have enough processor time for some reason to operate properly.
I thoroughly checked communication between php-fpm and redis with the help of network sniffer. Redis received request over tcp but sent the answer back only after one and a half seconds. It obviously signified that the problem is inside redis, that it cannot process so many requests in the given conditions (possibly processor starvation as the processor was only 50% loaded for the whole system).
The problem was resolved by moving redis to other server that was nearly idle. I suppose that we should have played with linux scheduler to make it work on the same server, but have not done it yet.
Bear in mind that Redis is single-threaded. If the operations that you're doing err on the processor-intensive side, your requests could be blocking on each other. For instance, if you're doing HVALS against hashes with very large values, you're going to make all of your clients wait while you pull out all that data and copy it to the output buffer.
Part of what you need to do here (regardless if this is the issue) is to look at all of the commands that you're using and determine the complexity of each command. If you're doing a bunch of O(N) commands against very large amounts of data, it's not impossible that you're simply doing too much stuff at a time.
TL;DR Nobody on here can debug this issue with real certainty without knowing which commands you're using and what your data looks like. But you can look up the time complexity of each method you're using and make sure it's reasonable.
I ran across this in researching an issue I'm working on but thought it might help here:
https://groups.google.com/forum/#!topic/redis-db/uZaXHZUl0NA
If you read through the thread there is some interesting info.