Posted on: 16/02/2012 22:56
More Gearman investigating... Try as I might, my idea for killing the worker just isn't happening (nicely). I've launched 5 workers (using supervisord - see my previous post), in my example (source code on Github), the most I can manage seems to be restarting the job (with another worker), but with "sort of" killing the job workaround hackyness.
From what I can gather this is what is happening:
- Worker A - starts getting weather (Job A - get_weather)
- Worker B - Kill Worker A (Job B - kill_job)
- Worker C - starts getting weather (Job A - get_weather)
You get the picture I think... Basically even though the killing works and Worker A is killed, Gearman is just so efficient it pops the job onto a new worker to carry out, leaving me high and dry as to how to simply get rid of that job, even if it's mid-processing. That's a killer feature I'd love to have. I understand why Gearman does it this way, but I really want to find a way of killing the worker AND the job (properly) so that particular job never happens again.
Having said all that, I have found a very hacky workaround. Basically, it goes a little something like this:
- Worker A - starts getting weather (Job A - get_weather)
- Worker B - Kill Worker A (Job B - kill_job) also write a file called /tmp/gm_norestart_job_a (using the job handle)
- Worker C - Checks for /tmp/gm_norestart_job_a - if it exists, return immediately. If not, start getting weather.
So basically when killing a job, place an empty file in /tmp to say "don't restart this job!". There are many drawbacks to this method, the most glaring is the fact this wouldn't work across multiple worker servers. But the concept is there! The code as I said is on Github, feel free to check it out... or not.
Posted on: 16/02/2012 22:12
Did a bit more playing with Gearman following on from yesterday's blog post. It turns out that supervisord is a pretty good bet for running our PHP workers as daemons in Linux.
sudo apt-get install supervisor
sudo nano /etc/supervisor/conf.d/gearman-test-worker.conf
In that file put these contents:
[program:gearman-test-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php worker.php
numprocs=5
directory=/home/james/workspace/gearman-testing
stdout_logfile=/var/log/gearman-job-server/supervisord.log
environment=GEARMAN_USER=gearman
autostart=true
autorestart=true
user=gearman
stopsignal=KILL
I had some issues with "restarting" supervisord, so instead I then issued:
$ sudo /etc/init.d/supervisor stop
Stopping supervisor: supervisord.
$ sudo /etc/init.d/supervisor start
Starting supervisor: supervisord.
$ ps ax
PID TTY STAT TIME COMMAND
1 ? Ss 0:00 /sbin/init
2 ? S 0:00 [kthreadd]
3 ? S 0:00 [migration/0]
... SNIP ...
4376 ? Ss 0:00 /usr/bin/python /usr/bin/supervisord
4380 ? S 0:00 /usr/bin/php worker.php
4381 ? S 0:00 /usr/bin/php worker.php
4382 ? S 0:00 /usr/bin/php worker.php
4383 ? S 0:00 /usr/bin/php worker.php
4384 ? S 0:00 /usr/bin/php worker.php
4403 pts/0 R+ 0:00 ps ax
As you see, this configuration file is set to launch 5 worker processes. If you tail /var/log/supervisor/supervisord.log, you should see something like:
2012-02-16 20:51:52,079 INFO supervisord started with pid 4376
2012-02-16 20:51:53,083 INFO spawned: 'gearman-test-worker_00' with pid 4380
2012-02-16 20:51:53,086 INFO spawned: 'gearman-test-worker_01' with pid 4381
2012-02-16 20:51:53,088 INFO spawned: 'gearman-test-worker_02' with pid 4382
2012-02-16 20:51:53,090 INFO spawned: 'gearman-test-worker_03' with pid 4383
2012-02-16 20:51:53,097 INFO spawned: 'gearman-test-worker_04' with pid 4384
2012-02-16 20:51:54,194 INFO success: gearman-test-worker_00 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2012-02-16 20:51:54,194 INFO success: gearman-test-worker_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2012-02-16 20:51:54,194 INFO success: gearman-test-worker_02 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2012-02-16 20:51:54,194 INFO success: gearman-test-worker_03 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2012-02-16 20:51:54,194 INFO success: gearman-test-worker_04 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
If you kill a worker process, or it does, supervisord will automatically restart it (autorestart=true in the .conf):
$ sudo kill -9 4383
$ tail /var/log/supervisor/supervisord.log
.. SNIP ..
2012-02-16 21:00:29,501 INFO exited: gearman-test-worker_03 (terminated by SIGKILL; not expected)
2012-02-16 21:00:30,504 INFO spawned: 'gearman-test-worker_03' with pid 4765
2012-02-16 21:00:31,667 INFO success: gearman-test-worker_03 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
$ ps ax | grep worker.php
4380 ? S 0:00 /usr/bin/php worker.php
4381 ? S 0:00 /usr/bin/php worker.php
4382 ? S 0:00 /usr/bin/php worker.php
4384 ? S 0:00 /usr/bin/php worker.php
4765 ? S 0:00 /usr/bin/php worker.php
4874 pts/1 S+ 0:00 grep worker.php
As you can see, there are still 5 workers running! Handy stuff.
Note to self... I could potentially use this pid to kill "broken" tasks. I want the ability to kill a task that has been running too long, interrupting it and losing any data. The steps could be something like:
- When worker starts a job, write to database a map from job handle (e.g. H:hostname:21) into server (192.168.0.5) and pid (4765)
- If we want to kill that particular job, we send another job out to say "kill H:hostname:21" and the worker will kill that pid on that server.
I'm sure another blog post will follow! :)
Posted on: 15/02/2012 21:26
I had a fiddle around with Gearman today, not entirely sure why, I just wanted to. But that's the sort of thing I like doing, so get over it.
This isn't a blog post as such, more of a list of stuff I did to make it work in Ubuntu 10.04, because the Gearman in even the offical Gearman PPA is 0.14 which is old, and the PECL Gearman library doesn't compile with it nicely. Find the latest version from https://launchpad.net/gearmand. At the time of writing, that's 0.28.
sudo apt-get install libboost-all-dev
cd /usr/local/src
wget http://launchpad.net/gearmand/trunk/0.28/+download/gearmand-0.28.tar.gz
tar zxvf gearmand-0.28.tar.gz
cd gearmand-0.28
sudo ./configure
sudo make
sudo make install
sudo pecl install gearman
There's an assumption here you've got stuff like the compiler etc. installed. I didn't have libboost, so I had to install that. The "sudo ./configure" will tell you what you need to install.
Once I got it working, I tested it with some example code I wrote/adapted from other examples (a good one was Playing with Gearman). My Gearman testing code is available on Github. In one window I had:
$ php worker.php
Waiting for job...
Handle job: H:shinylaptop-ubuntu:16
Adding.. 0 + 5 + 3 + 6 = 14
Waiting for job...
And in the other I did:
$ php client.php
Task complete...
string(2) "14"
$
It's an interesting concept, and I think I could put this into practice in appropriate places and I'm working on learning a bit more about it.
Update... I have since pretty much invalidated the examples in this post by updating my Github testing code, but maybe I'll write a new post soon with more info. I've got client-blocking.php and client-nonblocking.php which have examples of two methods of issuing jobs - there's quite a few ways of doing it, so go experiment! It was a good evening of refreshing hacking for me :) Link to my updated testing code.
Posted on: 28/11/2011 09:22
For me it didn’t seem very intuitive to find out what packages were installed when I tried removing the phpUnderControl PEAR channel (I trialled it but preferred Jenkins). This tutorial has nothing personal against phpUnderControl, it just happened to be the package I was removing at the time... All I got was:
$ sudo pear channel-delete phpuc
Channel "pear.phpundercontrol.org" has installed packages, cannot delete
$
Oh ok... maybe I should try:
$ pear list pear.phpundercontrol.org
parsePackageName(): invalid package name "pear.phpundercontrol.org" in "pear.phpundercontrol.org"
$
Hmmf... after a bit of facepalming I finally discovered that `pear help ` existed, and the correct syntax to find installed packages for a channel is:
$ pear list -c phpuc
Installed packages, channel pear.phpundercontrol.org:
=====================================================
Package Version State
phpUnderControl 1.0.0 stable
$ sudo pear uninstall phpuc/phpUnderControl
uninstall ok: channel://pear.phpundercontrol.org/phpUnderControl-1.0.0
$ sudo pear channel-delete phpuc
Channel "pear.phpundercontrol.org" deleted
$
PEAR 0, James 1.
Posted on: 26/11/2011 09:29
I hardly have time to play video games these days, but the few times between Freyja running me round, the dog licking me to death, and the odd bit of open source contributions, I like to play good games that I can engage in, but not have to pour my whole life into playing. Unfortunately due to my lifestyle, this rules out a multitude of games I'd like to play (or at least check out), such as MMORPGs that generally require a subscription - of which £5 a month normally I can't even justify. Recently the only times the 360 has been powered up is when friends have come over and we've had a bit of a Call of Duty shoot-up. This review does come over a year after the original release, but I'm still only part way through playing it.
Nevertheless, when I do have some spare time I love sitting down and playing Fallout: New Vegas. I loved Fallout 3 and played it for hours. In fact I was still playing it when Hannah went into labour with Freyja and still loved it. I still haven't played it as much as I'd like to and there's still so much to explore and discover in the game, even though I've already completed it.
Fallout: New Vegas does not disappoint me. From the word go it was already better - the character creation sequence is 5 minutes or so rather than the lengthy drawn-out character creation process in Fallout 3. Much improved - thank you Bethesda for letting us just get on with playing the game.
You start the game dead (as good a place as any I guess?) and if you're used to Fallout 3 everything will be pretty familiar. There are new additions such as temporary skill magazines (which confused me at first - why are my skills disappearing after a while? Don't fret though, the permanent skill books are still there, but even harder to find!), and the ability to play in Hardcore mode. I'm not hardcore, so I didn't do this, but from what I hear it makes you have to eat and drink and sleep just like a real person would have to do amongst other things. I can see that getting annoying after a while, but I suppose for those with plenty of time on their hands can become all part of the challenge. Good to see it's optional though!
There are some monsters which make a comeback (such as the practically invincible-to-a-new-player Radscorpions - why are they so hard to kill?!) and some new breeds and varieties that will shake things up a bit. The whole storyline is obviously based around New Vegas and I spent quite some time not being anywhere near New Vegas, but trying to get there without being slain instantly by an angry swarm of Deathclaws at Quarry junction. Nevertheless the story takes a real engaging turn when you get to New Vegas and has been really gripping, typical of Bethesda games.
Overall I think the game is slightly more difficult (and I'm playing it in "casual" mode, where food/water/sleep etc. are not a strict requirement!) than Fallout 3. The storyline is just as strong strong, but disappointingly there are very few graphical improvements on Fallout 3. As soon as I complete this, I'll be onto Skyrim to take in the delights of the new engine and the next TES game. However judging from my progress in New Vegas so far, I don't think that'll be for another year or two yet!
My overall rating for F:NV? I'd give Fallout 3 a 9 our of 10 and this is probably a 8 out of 10 due to only minor improvements, and much of the same stuff.
Please read more of my blog over on www.asgrim.com!