Premature Optimization

In the last couple of days the PHP world is boiling with excitement, with discussions about namespaces implementation in PHP 6.0. In my non-internals eyes, Dimitry’s proposal looks both simple and elegant.

I have very little to say beyond what was already said by others - but there is one thing I feel needs to be said: A feature like namespaces is important and expected enough, and should be implemented even if it means postponing the release of PHP 6.0. Most PHP users didn’t migrate to PHP 5.x yet - so while they do that, both the planned Unicode support and the emerging namespaces suggestion can be implemented into PHP 6.0 - even if this means the release will be delayed.

In other news, it looks like I’ve just been accepted to speak at the Zend Conference in October 07. Some people might thing this is no big deal if you work for Zend, but looking at the other speakers, and at the number of people who proposed topics, this is quite an honor for me just to be considered.

Right now I plan to deliver two talks: One which I already gave and IPC07, about Zend_Search_Lucene. The second, which will be more manager / team-leader oriented, will be about building scalable development environments - and in scalable the I mean procedure, infrastructure and people wise, and not performance / technology wise. So it’s going to be an interesting talk I hope - and not the regular technical mumbo-jumbo I usually get to talk about.

In any case the conference itself is probably going to be loads of fun and I’m really looking forward to it.

Happy times and great success! 1.0 is finally out, and I’m sure it will (already has?) got quite allot of coverage in the PHP blogsphere - and even got into the main page of Digg - so I’m not going to write about the framework too much - however I do feel a personal retrospection is in place…

I joined the ZF community back in May 2006 (at least this is the earliest SVN commit I could find) - I made some complaints about the HTTP client, and Andi suggested I take charge of it. One thing led to another, and I ended up rewriting the whole Zend_Http classes. Things were not always done the “framework” way (there were no proposals or specs up until later in the development process for example), but I think the result is decent and useful.

There is already lots of talk about the next stages - we’ve been in a feature freeze for quite some time, and I have plans for several requested changes and new components: It seems that in the last couple of weeks the growing interest got people to report bugs and ask for interface changes - ironically at the stage when it was to late to make any changes. So there is already a list of proposed improvements to Zend_Http classes - I plan on doing some work on Zend_Http_Cookie and Zend_Http_CookieJar - and perhaps one day also improve Zend_Http_Client doing things like creating a Zend_Http_Request class that will be inter-operable with Zend_Controller_Request_Http. Also, there is the much needed support for CONNECT requests for HTTPS proxying - and more.

Additionally, I put up (and am still updating) a proposal for Zend_Service_PayPal - a PayPal web services component. Hopefully this proposal will be accepted soon, and the pieces of code I already have can be moved from the laboratory to the incubator, and from there to the main library.

So the future of Zend Framework begins today, and 1.0 is definitely not the end - in fact, I have a feeling this might just be the beginning of, well urm.. a beautiful friendship

I just committed Zend_Service_PayPal to the Zend Framework laboratory. It’s basically a partial implementation of the PayPal NVP interface - and was committed so that people could enjoy what I already have written, as well as provide feedback, before the proposal is accepted (or rejected).

You can check the laboratory out from Subversion here: http://framework.zend.com/svn/laboratory/ - be sure to post any feedback either in the fw-webservices@lists.zend.com mailing list, in the proposal page on actually, here.

Today I learned about the (relatively new) realpath_cache_size and realpath_cache_ttl php.ini directives, which were added in PHP 5.1.0.

PHP 5.1.x and up caches the real path of include files, thus avoiding the extra system call when include_once and require_once are used (or at least, this is what I understood - hope I didn’t confuse anything).

Apparently, the cache size is configurable, and if you’re system has lots of RAM, you can increase realpath_cache_size - the default is 16Kbyte, which means you’ll have 16Kbyte of cache per Apache process. Almost every decent production system can handle an additional 16KBytes per process, and as I understand, this could have a positive effect on performance in applications that have lots of include files.

IPC07 is not over yet, but my talk is - it was quite nice IMHO, I got lots of questions, and ran quite a bit over my time - but since it was the last talk of the day it was Ok I guess.

I think that my main goal, which was to show how easy it is to start indexing existing content, was acheived and several people came to me and said they thing Zend_Search_Lucene is one killer component.

The slides are available here.

Looks like I’m going to be at the coming up International PHP Conference in Stuttgart. In fact, it looks like I’m going to speak about Zend_Search_Lucene. Not going to be a too advanced talk, I’m going to go over the basics of using Zend_Search_Lucene to index your existing content - never the less I think it’s quite interesting because Lucene makes it possible for even small sites to easily index their content and provide great search capabilities, and it’s incredibly simple to use.

If you’re there, come over and say hi

I was doing some work optimizing the memory usage and performance of Apache and PHP. Along the way, I hacked this tiny CLI PHP script (which in turns uses awk, grep, ps, xargs and pmap) to find out the average private memory consumption of all Apache processes. Very useful when you want to see how much RAM your binaries eat before and after handling requests (they tend to grow, that is why MaxRequestsPerChild is so important…)

You can get the script here. It can be executed either by chmodding it executable (chmod a+x) and running it (if you have a PHP binary in $PATH) or by running `php ./memusage.php`. It might give more accurate results when executed as root, due to the nature of pmap and memory access restrictions.

You might also have to change the value of $apache_proc to match your Apache binary name (for example on Debian and Gentoo that might be ‘apache2′ instead of ‘httpd’).

It works by finding the PIDs of all child Apache processes, running them through `pmap -d` and averaging the results. I bet an experienced bash hacker could have done it all in one line with no PHP wrapper, but I just couldn’t find a way to convert the output of pmap into proper numbers (without the “K” suffix), so it was easier to do the math in PHP.

Needless to say, this is a *nix script, and was tested with Debian Linux. If you’re a Windows user, you should start looking for a $50 nagware that does the same, or even better, start migrating now

I had an interesting conversation with Nir Yariv a couple of weeks ago in which we got to talk the meaning of the well-known (and probably my favorite) software development moto: “Premature optimization is the root of all evil”, coined by C. A. R. Hoare, and the special meaning it has in modern web development environments.

In my opinion, “Premature Optimization” is the very common practice of spending too much time optimizing something before we even know it needs optimization. Many developers, including myself, tend to seriously try to optimize things even when it’s not cost-effective or productive, just because we want to do things elegantly. But when exactly is optimization not premature? And what does it mean specifically in web development? That’s an interesting question, and a sort of question us techies tend to neglect too often.

I meet with lots of professional web development teams, in companies ranging from the smallest shops and startups to the largest web enterprises world wide. Almost all of them have something in common: They have no time. They all work in a market-driven (and too often marketing-driven) environments: priorities change daily and new feature requests keep coming in with very little quiet periods. There is no time (and sometimes no will from the management) to fix bugs, and especially no time to optimize. Beyond that, most of the applications they develop go through a different life-cycle than traditional software release cycles: the application is built, released once, and then (perhaps due to lack of time / resources / bad release management) it is patched and patched and patched. The application never goes through a second “release” - so when should the optimization phase be performed?

I’ve been thinking about this allot, and I came up with an interesting conclusion: “Premature Optimization” in web applications mean focusing on performance instead of on maintainability, modularity, flexibility and generally knowing your code well. I’m not saying performance is not important: you should develop with performance in mind. But it should not come at the expense of maintainability and modularity - because those will allow you to both fix bugs and scale easily when the time comes.

These thoughts are not revolutionary, but I think they make an important note, especially today when code generation is so popular, PHP source code to C extension conversion pops up again, and claims that source code documentation is overrated are heard (here and here).

Having this conclusions in mind, I came up with a short (partial) list of tips for people who start writing a web application and want to know where to put their focus:

• Write maintainable code - in other words, decide on coding standards and follow them. In many cases it also means go OOP - but not always.
• Write modular code - Yes, OOP again, but I think that in modern web applications it comes down to one important concept: MVC. You don’t have to use a bloated MVC framework - but separate your tiers. It’s worth it.
• Document - Being able to scale means being able to bring in new developers that will be able to jump into the water as soon as possible. The more in-line documentation you’ll have, the easier it will be for them to both understand how the application works and learn from other people’s experience.
• Keep it Simple - If you’re going with a framework, make sure you know it well enough to hack it when it comes to it. Make sure you create (or use) a modular framework, that allows you to decouple tasks. Do things the UNIX way - small and simple tools that perform a single task and perform it well.
• Be Careful with Code Generation - Again, just like with frameworks, ode generation is a double-edged sword. Too much of it - and you won’t know what your application does. If you use some auto-generator, make sure you study it’s output well.

I’m sure this list can be extended, but my main point is - don’t over do things. Do things in a way you’ll be able to easily improve them when you need to, and release on time. You’ll thank yourself when you’re slashdotted.

It’s 4:30 AM, and I’m sitting in an airport lounge waiting for a flight to Brussels after accidentally bumping into Zeev and a couple of other guys (Amnon and Matti) from Zend, on their way to a management meeting in the US. Since they had to board, and I’m left all alone and still have an hour or so to pass, I’m going to share a nifty little thing I found in Zend Framework a couple of days ago: A nice and clean way to generate relative links in view scripts.

[more...]