Running multiple processes in PHP

May 25th, 2010 19 Comments

Sometimes you need multiple commands to run in parallel to save up script processing time for repetitive tasks.
In the “Building a Video sharing site” project that i will present here soon, I needed a script to run multiple video processing jobs. PHP wasn’t meant to support multitasking, but with a few tricks we can emulate a multitasking environment.

On my research there were a couple of solutions that, althought they might seemed viable, had some weakness:

Using curl to open multiple instances of the script at once
The problem was that the script was supposed to be accessible via the Web, and when trying to access external commands, this is a big security issue.
Using pcntl functions
This could have been the best approach, but it required recompiling the PHP version with –enable-pcntl option, and many of us don’t have access to recompile PHP on the server. Beside that, it only works on linux systems, so it’s platform dependent.
Finally i found a nice class that simulates multi-threading and after some customisation i managed to set the multi tasking the way i wanted:

Threads.php

<?php
class Thread {
	var $pref ; // process reference
	var $pipes; // stdio
	var $buffer; // output buffer
	var $output;
	var $error;
	var $timeout;
	var $start_time;
 
 
	function Thread() {
		$this->pref = 0;
		$this->buffer = "";
		$this->pipes = (array)NULL;
		$this->output = "";
		$this->error="";
 
		$this->start_time = time();
		$this->timeout = 0;
	}
 
	function Create ($command) {
		$t = new Thread;
		$descriptor = array (0 => array ("pipe", "r"), 1 => array ("pipe", "w"), 2 => array ("pipe", "w"));
		//Open the resource to execute $command
		$t->pref = proc_open($command,$descriptor,$t->pipes);
		//Set STDOUT and STDERR to non-blocking 
		stream_set_blocking ($t->pipes[1], 0);
		stream_set_blocking ($t->pipes[2], 0);
		return $t;
	}
 
	//See if the command is still active
	function isActive () {
		$this->buffer .= $this->listen();
		$f = stream_get_meta_data ($this->pipes[1]);
		return !$f["eof"];
	}
 
	//Close the process
	function close () {
		$r = proc_close ($this->pref);
		$this->pref = NULL;
		return $r;
	}
 
	//Send a message to the command running
	function tell ($thought) {
		fwrite ($this->pipes[0], $thought);
	}
 
	//Get the command output produced so far
	function listen () {
		$buffer = $this->buffer;
		$this->buffer = "";
		while ($r = fgets ($this->pipes[1], 1024)) {
			$buffer .= $r;
			$this->output.=$r;
		}
		return $buffer;
	}
 
	//Get the status of the current runing process
	function getStatus(){
		return proc_get_status($this->pref);
	}
 
	//See if the command is taking too long to run (more than $this->timeout seconds)
	function isBusy(){
		return ($this->start_time>0) && ($this->start_time+$this->timeout<time());
	}
 
	//What command wrote to STDERR
	function getError () {
		$buffer = "";
		while ($r = fgets ($this->pipes[2], 1024)) {
			$buffer .= $r;
		}
		return $buffer;
	}
}
 
 
//Wrapper for Thread class
class Multithread{
	var $output;
	var $error;
	var $thread;
	var $commands = array();
 
	function __construct($commands){
		$this->commands = $commands;
 
		foreach ($this->commands as $key=>$command){
			$this->thread[$key]=Thread::create($command);
		}
	}
 
 
	function run(){
		$commands = $this->commands;
		//Cycle through commands
		while (count($commands)>0){
			foreach ($commands as $key=>$command){
				//Get the output and the errors
				$this->output[$key].=$this->thread[$key]->listen();
				$this->error[$key].=$this->thread[$key]->getError();
				//Check if command is still active
				if ($this->thread[$key]->isActive()){
					$this->output[$key].=$this->thread[$key]->listen();
					//Check if command is busy
					if ($this->thread[$key]->isBusy()){
						$this->thread[$key]->close();
						unset($commands[$key]);
					}
				} else {
					//Close the command and free resources
					$this->thread[$key]->close();
					unset($commands[$key]);
				}
			}
		}
		return $this->output;
	}
}
?>

I call the class using the following code:
Example.php

<?php
set_time_limit(0);
include "threads.php";
$commands = array('ffmpeg -i '.$inputFile[0].' '.$outputFile[0].' 2>&1','ffmpeg -i '.$inputFile[0].' '.$outputFile[0].' 2>&1');
$threads = new Multithread($commands);
$threads->run();
foreach ($threads->commands as $key=>$command){
	echo "Command ".$command.":<br>";
	echo "Output ".$threads->output[$key]."<br>";
	echo "Error ".$threads->error[$key]."<br><br>";
}
?>

Let’s talk about what the script does:

First in Create method it defines some descriptors to use with proc_open. The standard descriptors for shell commands are:
0 for stdin (what is read by the process)
1 for stdout (what is the output of the runing process)
2 for stderror (errors that the process throw).
You can customize your settings here, for example you can choose to log all the errors in a file like this:

2 => array("file", $path_to_error_log_file, "a")

By setting stream_set_blocking($t->pipes[1],0), and stream_set_blocking($t->pipes[2],0) we tell php to not wait for the process to finnish with the output and error reporting, before runing another command. This is an important step, in achieving multitasking.
The current state of the runing process is achieved using isActive method, that will read from the stdout (stream_get_meta_data($this->pipes[1])) to see if the process is still active or not.
The command output will be generated by the $t->listen method. Aditionaly, you can capture the errors by calling getError method.
Close method will close the command strem and free the resources associated.
isBusy method will check to see if the $timeout time was passed. I recomend setting timeout high, if you know that your script take longer time to load. If timeout is set to 0 it will not timeout any script.

In the attached file you will see an example implementation of this technique. 3 scripts (one.php, two.php, three.php) will write data in a log file with one second delay between writings, to see how the multitasking is achieved, err.php will simulate a script producing an error.
Have fun experimenting with the script, and if you have any questions, just leave a comment here.

Download code for this post

No related posts.

Posted in PHP | Tags: Multithreading, PHP, proc_open

19 Responses to “Running multiple processes in PHP”

masterb says:

December 4, 2010 at 12:00 am

Great script, works perfect!
Nickolas Wood says:

January 6, 2011 at 8:37 pm

Hi,
I have modified your class a little bit and fixed a couple bugs for you; figured I would contribute back. First off, line 107 and 108 of your threads.php file return undefined reference notices when ini_set(‘display_errors’, ‘On’); and error_reporting(E_ALL); are used. Secondly, the thread timeout of 0 on line 20 of the threads.php file (with no dynamic way of changing it) causes weird behavior when you start checking each thread for null output and resubmitting the command to a new multithread object on such null output. If a thread takes longer then 0 seconds to complete, the isbusy routine kills it. Most commands take longer then 0 seconds to return anything so I kept getting null output. This threw me for a loop for awhile.

These three problems have been fixed in the below code. I have also made a couple of minor changes to better handle output. Look for lines with #MODIFIED in them, those I changed. I hope someone else finds it useful.

Great work on your multithread class by the way. It helped me alot.

START CODE BLOCK:

pref = 0;
$this->buffer = “”;
$this->pipes = (array)NULL;
$this->output = “”;
$this->error=”";

$this->start_time = time();
$this->timeout = $timeout; #MODIFIED, used $timeout
}

function Create ($command, $timeout) { #MODIFIED, added $timeout
$t = new Thread($timeout); #MODIFIED, used timeout
$descriptor = array (0 => array (“pipe”, “r”), 1 => array (“pipe”, “w”), 2 => array (“pipe”, “w”));
//Open the resource to execute $command
$t->pref = proc_open($command,$descriptor,$t->pipes);
//Set STDOUT and STDERR to non-blocking
stream_set_blocking ($t->pipes[1], 0);
stream_set_blocking ($t->pipes[2], 0);
return $t;
}

//See if the command is still active
function isActive () {
$this->buffer .= $this->listen();
$f = stream_get_meta_data ($this->pipes[1]);
return !$f["eof"];
}

//Close the process
function close () {
$r = proc_close ($this->pref);
$this->pref = NULL;
return $r;
}

//Send a message to the command running
function tell ($thought) {
fwrite ($this->pipes[0], $thought);
}

//Get the command output produced so far
function listen () {
$buffer = $this->buffer;
$this->buffer = “”;
while ($r = stream_get_contents($this->pipes[1])) { #MODIFIED, replaced fgets with stream_get_contents
#MODIFIED, removed $this->output appending (unneeded)
$buffer .= $r;
}
return $buffer;
}

//Get the status of the current runing process
function getStatus(){
return proc_get_status($this->pref);
}

//See if the command is taking too long to run (more than $this->timeout seconds)
function isBusy(){
return ($this->start_time>0) && ($this->start_time+$this->timeoutpipes[2])) { #MODIFIED, replaced fgets with stream_get_contents
$buffer .= $r;
}
return $buffer;
}
}

//Wrapper for Thread class
class Multithread{
var $output;
var $error;
var $thread;
var $commands = array();

function __construct($commands, $timeout){ #MODIFED, added $timeout
$this->commands = $commands;

foreach ($this->commands as $key=>$command){
$this->thread[$key]=Thread::create($command, $timeout); #MODIFIED, used $timeout
$this->output[$key] = “”; #MODIFIED, added this line to remove undefined reference notices
$this->error[$key] = “”; #MODIFIED, added this line to remove undefined reference notices
}
}

function run(){
$commands = $this->commands;
//Cycle through commands
while (count($commands)>0){
foreach ($commands as $key=>$command){
//Check if command is still active
if ($this->thread[$key]->isActive()){
//Get the output and the errors
$this->output[$key].=$this->thread[$key]->listen(); #I removed this line on my server as I don’t need it, I want all output or nothing. I could see where it is useful however
$this->error[$key].=$this->thread[$key]->getError(); #I removed this line on my server as I don’t need it, I want all output or nothing. I could see where it is useful however
//Check if command is busy
if ($this->thread[$key]->isBusy()){
#MODIFIED, removed $this-output (unneeded, if we kill a process then its output is unusable and there is no need to update it further)
$this->thread[$key]->close();
unset($commands[$key]);
}
} else {
//Close the command, gather results and free resources #MODIFIED, added, gather results comment
$this->output[$key].=$this->thread[$key]->listen(); #MODIFIED, added this line (want to make sure we have all output before we close a thread)
$this->error[$key].=$this->thread[$key]->getError(); #MODIFIED, added this line (want to make sure we have all output before we close a thread)
$this->thread[$key]->close();
unset($commands[$key]);
}
}
}
return $this->output;
}
}
?>

END CODE BLOCK:

I know, the formatting sucks but I couldn’t find a nice way of doing it. Now, I call this class like so:

START CODE BLOCK:

$threads = new Multithread($command_list_exec, $execution_time);
$threads->run();

END CODE BLOCK:

As I mentioned earlier, I check each process’s output against several strpos instances for known conditions. The most common problem I saw was no output at all. If this was the case then it was killed before anything had a chance to be reported and it needs to be run again.

It is vitally important that set_time_limit($execution_time); is used in php pages that call the multithread class so that a rogue process doesn’t cripple a with endless thread creations/kills.

Hope this helps.

Thanks again!!
Nickolas Wood says:

January 6, 2011 at 8:40 pm

I apologize, I missed a couple lines of code at the very top; at the first START CODE BLOCK. This codes should be inserted there:

START CODE BLOCK:

END CODE BLOCK:
Nickolas Wood says:

January 6, 2011 at 8:41 pm

Umm, let try this again removing the initial php tag:

START CODE BLOCK:

class Thread {
var $pref ; // process reference
var $pipes; // stdio
var $buffer; // output buffer
var $output;
var $error;
var $timeout;
var $start_time;

function Thread($timeout) {
$this->

END CODE BLOCK:

This will be my last attempt so that I don’t over post here.
Silviu says:

June 12, 2011 at 11:06 am

Hello. I’ve tried your code and I get the following error. Please help

“Error ‘tasks’ is not recognized as an internal or external command, operable program or batch file.”
carlos Pimentel says:

August 2, 2011 at 1:19 pm

Hi Nickolas Wood,
could you please post the code that you’ve created in pastebin.com or similar website ?
This code isn’t working for me, sometime I receive the output from the process some time I don’t and I almost crazy debugging this…
Thanks !
bee says:

December 5, 2011 at 1:49 pm

Hey cool threads.php class, Works great. I got a question anyway, what is the maximum number of scripts that can be multitasked?

Thanks,
bee.
www.hizlandirma.com says:

April 29, 2013 at 9:52 pm

…Trackback: More Informations on that topic…

[...]Excellent weblog right here! Additionally your site quite a bit up very fast![...]…
top eleven hack no survey no password says:

June 28, 2013 at 11:03 pm

Hi there! This is my 1st comment here so I just wanted to
give a quick shout out and say I genuinely enjoy reading through your articles.

Can you suggest any other blogs/websites/forums that cover
the same subjects? Many thanks!
jgyuejdifiofersawhp says:

November 26, 2013 at 2:39 am

Extraordinaires Fendi Artiste Totes

Si vous dÃ©cidez de diffuser Ã la suite de l’artÃ¨re Santee de Shedd Angeles et de plus rÃ©centes Canal Avenue de York, vous remarquerez sosie sacs Ã sa disposition sur la route. Faux bacs pourraient trÃ¨s bien Ãªtre Ã l’aise grÃ¢ce Ã l’utilisation de sa mousseline Ã©lectronique, longchamp taschen reparatur bon achat adressage et logos d’art par ailleurs mal orthographiÃ©s sont definitily un Ã©lÃ©ment. S’il vous arrive d’apercevoir le fait, comment Ã§a va raboter un voyage Ã choisir? Le problÃ¨me est gÃ©nÃ©ralement que un autre sac Louis Vuitton est sans doute trop cher et ne devrait pas finir par Ãªtre fourni en utilisant un individu normal. Un seul coup d’oeil longchamp taschen ratingen dans le coÃ»t ainsi que les espoirs d’une personne devrait recevoir pÃ©ri. Les vrais Vuitton sacs Ã main Louis viennent Ã©galement dans quelques modÃ¨les assez, couleurs ainsi que le dimensionnement de Il accueille Ã peu prÃ¨s tous la dÃ©pendance unique sur absolument gratuit themes.Memorable vous faire des cheveux qui va gÃ©nÃ©rer son homonyme longchamp taschen roseau incroyablement, les styles de couleurs chics cependant animÃ©s prÃ©sentant aujourd’hui de chic must-haves. Pour Ãªtre honnÃªte, je revÃªts tout et tous les analystes. Ou Prenons votre Hobo Hysteria trouvÃ© dans insignes beige typique nÃ©anmoins couplÃ©e ces temps Ã une autre Ã mourir pour le matÃ©riel de gemme.

Hampton rapide pause mode Cette Ã©charpe Impress Demi profite Ã une poche ou un sac externe particulier, filez premier securer, marque Ã©charpe tissu de tissu par voie de daim et de plus digne cuir des mÃ©nages. Cette demi unique de 200 $ s’applique. Vous y trouverez des sacs Ã main fabuleux supplÃ©mentaires sur le net ou peut-Ãªtre en gÃ©nÃ©ral le store.Although ou mÃªme regardÃ© mÃ©thodes pour longchamp taschen rechnung Ã©conomiser votre argent en collectant des sacs Ã main mode reconnus? Semble Ãªtre extrÃªmement difficile? Il n’est tout simplement pas! La fonction sac hermes birkin est particulier avec le Jane Birkin idÃ©e prÃ©sentateur. confortablement personnalisable earn sac Birkin ou Kelly felix Ã©tuis de transport Ã l’intÃ©rieur par rapport Ã consulter mois. vous pourriez favoriser une nuance spÃ©cifique de couleur, chamois, des semaines DiamÃ¨tre intÃ©rieur, le travail intÃ©rieur de diamÃ¨tre thus.The dames ensemble de sac Ã main est considÃ©rÃ©e comme l’un bon nombre de gros sacs Ã main imitÃ©s en ces temps modernes. Une personne spÃ©ciale avancÃ© connectÃ© avec des sacs Ã main abordables vous devez savoir pour investir dans un sac Ã main rÃ©elle est connu pour devenir cette volontÃ© d’argent et plus de style. Ces jours-ci nous allons parler de la meilleure faÃ§on prÃ©fÃ©rÃ© ne pas mentionner souhaitable ce genre de sÃ©lection est.

extraordinaires Fendi Artiste emballages, les emballages et mÃªme les portefeuilles et sacs Ã main Ã©tonnants Fendi sacs Ã main, pochettes personnalisÃ©es abordables ainsi que des portefeuilles, Une touche longchamps taschen rot Fendi Le passÃ© historique Fendi est sÃ»rement un designer italien de votre maison la plus courante que pour les embrayages de “baguette”. Qu’il a Ã©tÃ© publiÃ© quand il s’agit de 1925 comme une peau sans parler dÃ©taillant d’alligator dans la capitale, Ã l’Ã©poque actuelle est vraiment un des produits de l’extravagance internationales et le nom de marque de solutions achetÃ©s Ã la suite de LVMH.Some trÃ¨s paresseux, tout en Ã©tant l’Ã¢ge Ã¢gÃ© et longchamp le pliage for sale in the philippines donc chic. Je ne suis pas en gÃ©nÃ©ral dans des cas extrÃªmement organisÃ©s, que je prÃ©fÃ¨re gÃ©nÃ©ralement une rÃ©cession plus pour vraiment mes propres backpags. Et c’est aussi sans doute prÃ©cisÃ©ment pourquoi mon groupe est excitÃ© au sujet de Proenza Schouler PS1 sacs Ã main! Vous pouvez aussi augmenter le risque pour les couleurs selon vos normes. Il ya beaucoup de couleurs pour prendre le sÃ©lectionner. Certaines options comprennent les fruits, violet, gris, noir colorÃ©, lumineux, Ã©cologique, bronzage lÃ©ger, en plus grisÃ¢tre.
ugg boots cheap canada says:

November 26, 2013 at 2:40 am

look into http://www.thedailymamma.com/category/45/ ,then mens ugg slippers show up
Canada Goose Jakke Dame Lang says:

November 26, 2013 at 2:40 am

Her er mye i forhold til canada goose i norge søk i http://www.babagroups.com/
シャネルミニバッグ says:

November 26, 2013 at 2:41 am

アディダスジェレミースコット格安販売http://www.back2iraq.com/アディダス-ジェレミー-スコット-c-85.html
モンクレール人気 says:

November 26, 2013 at 2:41 am

話題沸騰中のシャネル財布人気 http://www.11apcedm-bali.com/シャネル-財布-c-82.html
click here says:

December 17, 2013 at 3:53 am

Acrílico: contracción, sabor picante, ceniza negro duro quebradizo o esfera hueca.
basket basse says:

January 10, 2014 at 5:23 am

Taoka is really a excellent coach, but has made errors in judging his players and also the opposition. He nonetheless has an eye for expertise although, as he recognized Uozomi’s possible and gave him reassurance when Uozomi’s abilities were weak and he was considering of quitting. Taoka also recognized that when Sendoh and Fukuda joined the group they would grow to be crucial gamers.
basket basse http://www.brianjordanmusic.co.uk/nike-cortez.asp?id=basket-basse
louis vuitton speedy graffiti says:

January 10, 2014 at 5:52 am

Essentially the most reliable device professionals have for motivating call center agents could be the a single product that is certainly most frequently mentioned as missing when agents are surveyed. That device is recognition. Agents need to be acknowledged once they do a thing very effectively, of course, but it is not just a make a difference of acknowledging major performers..
louis vuitton speedy graffiti http://www.dfi-bpi.com/lv/louis-vuitton-speedy-graffiti.html
louis vuitton keepall bags says:

January 10, 2014 at 7:15 am

Team developing is a crucial work inside the continuing growth of any organization, no matter if brand name new or firmly established. A company’s employees are its perfect resources, and uniting them to feel, act, and function being a team is very important in almost any sector. There’s lots of completely different ways to make a team: interactive routines, skill growth workshops, and last although not minimum, only finding the time to acquire to know a single one other on an expert basis.
louis vuitton keepall bags http://www.strouddistrictcouncil.co.uk/lv/louis-vuitton-keepall-bags.html
louis vuitton pouch men says:

January 10, 2014 at 11:47 am

The 2 lived together briefly in his mansion in La, and continued a warm friendship even right after she still left his home. For his psychological, energetic performance, Sal was once again nominated for an Oscar. And again, Sal lost, the award instead planning to Peter Ustinov for his function in Spartacus.

PHP Code