My Abstraction Optimization

Here at Anus Health, you kinda get used to doing things multiple, unnecessary times: redoing the work of minions, retring bad data calls, or explaining complicated migration procedures to pointy haired bosses. The worst is writing code - write, rewrite with a better data model, reformat, add a tiny feature, clean - it never ends. With over 15,000 patient records filling over 13GB of data, today is the day where I need to start thinking about performance and rewriting better, more scalable code...

Abstract Out, Then Optimize In
I tend to build software according to the adage "if it doesn't make sense yet, just build another layer of abstraction". You know what? It works. Put time up front to make really nice abstractions and even the dumbest of colleagues are knockin' out working code like monkeys eat lice. But there is a price for abstraction in the end, and that tends to be performance. Every layer that you added is spinning the disk better than Punjabi MC and eating more RAM than your mom. The classic example of this is my good friend Plone. Amazing to quickly buildout an app, but I'll be damned if after 6 years of development experience, intense caching policies, and 16GB of RAM I still can't get it running much faster than a hobo on a train track.

In my last post, I talked a lot of shit about not worrying about scalability. If you're lucky, you get to the point where you actually have to start worrying because good caching and hardware is only gonna take you so far. The interesting thing about this is that after you spend all that time abstracting away from the nitty gritty, you know have optimize back down through the layers to find efficiencies: build things up and then tear them down. This is exactly what twitter is doing. Now that things are huge, start replacing those yummy developer abstractions with good old nasty C. Instead of using an ORM, write your reports direct from MySQL then take it a notch further and read directly from disk. Optimization high five - whipish!

The thing that uber sucks about optimizing an abstracted system is the fact that these optimizations are not easy to write. Current interfaces are entagled in kludges deeper than a whales vagina. The little pieces of backwords compatability from your 1034th version of the photo album picture are just barely holding up as it is. What is a developer to do? Have the Java weenies been right all along?!? Stay tuned...

Next time on Sincerely, Management:

• "My Responsibility Response" - Things get whackey at Anus Health when Liz learns that Plone can't do everything, especially reports. Guest starring "IT guy".
• "My Feed Lust" - After a losing battle with IP latency and unreliable 3rd party API responses, Liz shows instantaneous response the door. Not suitable for children under 25.
• "My New XML Fetish" - Liz sends JSON & GET packing after they discover her steamy affair with XML, starting a dangerous journey away from friendly 3rd party developer API land.