the { buckblogs :here } - Home

BucketWise v1.1.0

2009-05-15T03:55:00Z

So, I’ve now been using BucketWise for almost two months, and it’s been fantastic. Admittedly, as the author of the application, I’m willing to overlook a lot of the warts and inconsistencies, but I can honestly say I’ve felt more control over my finances these last two months than I’ve felt in the last 10 years. It’s an awesome feeling!

Tonight, I tagged version 1.1.0 of BucketWise, which (if you haven’t been following along) fixes a few bugs and adds several new features (account reconciliation, memorized transactions, actor name autocompletion, simple budget reporting, and more; I’ll just refer you to the changelog for the full list). It’s really been a fun project to tinker on. The last feature I myself really want is scheduled transactions; I may be hacking on that one in the near future.

I figured this might be a good time to talk a little about how I, personally, am using BucketWise. I’ve been surprised by a few things, both good and bad: some features I’ve found to be less useful than I anticipated, and others have been surprisingly handy!

Firstly, when you log into BucketWise, you see a short list at the top, called “Recent transactions”. This list was intended to let you see, at a glace, what you most recently had entered. (It also provided a handy landing place for newly entered transactions.) It hasn’t been very useful, though; I find that what I really want is to see the register of transactions for my checking account. I may be reworking that dashboard view soon.

Also, bucket reallocations haven’t been quite as useful as I expected. I do use them, and they are definitely handy, but I find that if you shuffle money around too much, it muddies your register. The reallocations are basically noise, especially when viewing transactions at the account level. I’m going to be pondering ways to reduce their visibility.

Buckets, though, I’ve found to be spectacularly useful. I’ve got my savings account partitioned into three buckets (short term, medium term, and long term), and that’s been a great way to keep track of how those savings funds are earmarked. Also, I’m trying to save 10% of each paycheck (trying, but not very successfully yet!), so I’ve got a “savings” bucket in my checking account, too. When the funds get to a certain threshold, I transfer the money to my savings account. (Yeah, I could just do a transfer with each paycheck…but I find I’m more likely to do it if I do it infrequently. Not sure why that is.)

Buckets are also great for indicating money that was given as a gift. My wife and I share the same checking account, so when it was her birthday, I transfered money from my Paypal account and put it in a “Tarasine” bucket. She was then welcome to record whatever purchases she wanted against that bucket. Similarly, when I receive money as a gift (birthday, Christmas, whatever) I just deposit it into a “Jamis” bucket.

Lastly, having credit card debt repayment built into the application has been awesome. I’ve loved being able to immediately indicate which checking account bucket a credit card purchase will be repaid from, and seeing that those funds are set aside, inviolate, ready for when the credit card bill comes.

My checking account currently has 35 buckets, and I can see my wife and I adding more. Most are purely for budgeting purposes (“groceries”, “auto fuel”, etc.), but they are so handy as ways to arbitrarily earmark money. Tithes, charitable offerings, savings, and credit card repayment are just some of the ways I’ve used them. (In fact, I’ve found myself wishing I could mark additional buckets as being “aside” buckets; I’m still pondering ways to make that happen, if it needs to.)

I’ll probably blog more about BucketWise down the road, and talk about specific use cases and how it’s helped me with them. However, I’d love to hear from others, too. Are you using BucketWise? If so, what do you like and dislike about it? I’m definitely only writing this application for me, but I’m curious to hear what the experience is like for others.

Lastly, if you’re interested in giving BucketWise a test drive, I’ve set up a demo account that you are welcome to log into and play with. I’ll reset the data there periodically, so feel free to try out all the features! Just go to http://www.bucketwise.com, and log in with the “bw.demo” user (password “demo”). Note that this is hosted on a modest Linode host, and will almost certainly be swamped into unusability with any significant traffic, but you’re welcome to try it out.

BucketWise v1.0.0

2009-04-20T13:36:00Z

BucketWise is now available! It’s far from “done”, but I’m labelling it “1.0.0” because giving code pre-1.0 version numbers is a coward’s game. (ha!)

At any rate, fork it on GitHub, see what you think, and contribute back if you feel so inclined. The TODO includes a list of unresolved issues and features I’d like to see land someday: knock yourselves out. :)

Note that I’ve tried to document the REST API for BucketWise, too; the first draft is in doc/API.rdoc let me know if any of that is too confusing or not informative enough (I’m sure that will be the case). I’m really looking forward to seeing what people use the API for.

Note that while the source code is now available, I’m only actually hosting the application itself for myself and a few close friends. But please feel free to deploy it to your own servers, or even just run it locally. And let me know what you think! BucketWise is super opinionated, so I’m sure I’ll get some hate mail, but that’s the name of the game when you’re playing things like this on the Internet. For me, personally, BucketWise has already proved more than worth the time I’ve invested in it so far; hopefully it’ll prove useful to others as well.

Enjoy!

BucketWise: Preview #2

2009-04-09T00:46:00Z

“Buckets” is now “BucketWise”. The name was more unique, easier to identify as an application, and just felt better than “Buckets”.

To celebrate the new name, I’ve also made another screencast, this one demonstrating BucketWise’s anti-debt features. It’s a bit more long-winded than the first one: five and a half minutes of me talk-talk-talking:

BucketWise Preview #2 (QuickTime, 5:30, 10MB)

I’m slowly working my way through the list of things that I want done before I release the code. For the curious, here’s what I’m wanting done before I open it up:

Existing bugs:

after saving a reallocation event, the line items don’t reset
green “your transaction has been recorded” notice doesn’t disappear when starting a new event
date column for event row is squashed in safari
“Add a new bucket” should set focus on new bucket line
check number and “select repayment options” links are out of whack when editing an event
deleting event from account perma results in ajax error (maybe only when there is only a single event in the account?)
Reallocating a bucket doesn’t change the updated_at field for the bucket?
“move in” and “move out” links on dashboard don’t use existing form—they go to events/new
make all autocompleting fields refresh their source array after event creation

Remaining features:

tag delete (optionally move transactions to different tag, e.g. “tag merge”)
tag rename
API
validations
rake tasks for user and subscription creation
change style for event detail view: “window shade” style isn’t really working
attr_protected everywhere (perhaps implement an update_override method or something for places where it would be useful to do mass updates despite protected attrs?)
spinner for ajax actions
user info page (for password change)
password reset from login page
cache balances on events (?)
README with installation instructions
CHANGELOG

Those lists are straight from my TODO file. But for those who think I’m waiting for too much to be finished, never fear. I’ve got plenty that I want to implement, but which I’m willing to wait until post-release to work on:

signup process
exception notifier (?)
/subscriptions index
autocomplete actor field in event form
better 404 and 500 error pages
searching & reporting
account reconciliation
transaction templates (“saved” or “memorized” transactions)
scheduled transactions (occur automatically at specified intervals)
show check numbers in event detail view
add a memo field for events
print stylesheet
oauth authentication for API
filter event views to show only events with a non-zero balance for the account in question
filter event views to show only deposits, or only expenses
normalize ‘actor’ so we can do actor-centric queries more efficiently
full-text searching on the memo field
show bucket and account balances next to names in drop-downs
make bucket reallocation work from bucket perma

Depending on how soon my bank sends the next statement for my checking account, I may or may not need to have “account reconciliation” done before launch, too.

As before, let me state: I’m writing this application for myself, and really only for myself. If other people like it, that’s cool. I’m more than happy to share and let other people hack on it. I may even release it into the public domain (I haven’t yet decided what license I’ll use). But BucketWise will always be very opinionated. If you prefer having your finance app download your transactions from your bank, you’ll be disappointed by BucketWise. If you need currencies other than the US dollar, you’ll be disappointed by BucketWise. If you need pretty charts and graphs to analyze your cash flow, or if you want to track investments, or any number of other things, you’ll be disappointed by BucketWise.

Just FYI. :)

Buckets: Preview

2009-03-24T14:07:00Z

So, yeah. With Capistrano and friends off my plate, I’ve actually found time to work on a project that has been in the works for years (and that’s no exaggeration, I first mentioned it in a blog post in October 2004). I’ve named it and renamed it (“Penny Pincher”, “Chump Change”, “Make Me Rich”, and “BudgetWise”) but its current incarnation is “Buckets”.

Buckets is a simple web-based personal finance application that I’ve been working on, written specifically for my wife and me. Its focus is on simple budgeting, and reducing debt, and It is intentionally “feature-poor”. It is loosely based on an envelope budgeting strategy, and while it definitely isn’t the only web-based finance app using such a strategy, it just may be the simplest.

I recorded a screencast demonstrating the budgeting aspect of Buckets; it’s a 5M QuickTime movie, 2:42 in length. Click here to view it, if you care to..

Buckets is still private: it has been deployed and my wife and I are using it, but that’s it. The source code is in a private repository on GitHub, and the production instance of the app is currently only accessible to me. That will change eventually (maybe a couple of weeks, depending on how initial testing goes), but I want to make sure it’s actually going to be useful before I open it up.

Net::SSH, Capistrano, and Saying Goodbye

2009-02-25T04:38:00Z

It is with mixed emotions that I announce two things this evening.

First, I’m announcing the final release of both Net::SSH (2.0.11) and Capistrano (2.5.5). Both are minor changes: Net::SSH 2.0.11 adds support for a :key_data option, so you can supply raw PEM-formatted key data. Capistrano 2.5.5 enhances the role() method so you can now declare empty roles. Either way, not much to get excited about, but the changes were pending and deserved releasing.

Secondly: I’m ceasing development on SQLite/Ruby, SQLite3/Ruby, Net::SSH (and related libs, Net::SFTP, Net::SCP, etc.) and Capistrano. I will no longer be accepting patches, bug reports, support requests, feature requests, or general emails related to any of these projects. For Capistrano, I will continue to follow the mailing list, and might appear in the #capistrano irc channel from time to time, but I am no longer the maintainer of these projects. I will continue to host the capify.org site and wiki for as long as they are of use to people.

This was a very hard decision, and one that has taken me months to come to grips with. I cannot express how much I appreciate the huge support from everyone that has found value in Capistrano, in particular. Your kind words and encouragement have meant a lot to me. But I’m burning out, and I have to drop these before things get worse. Maybe after some period of time I’ll come back to them—I don’t know. But I’m not planning on it.

So where do these projects go from here? That’s entirely up to the community. If you have a neat idea for any of these, please feel free to fork the project on GitHub (see my profile page for the links to the individual projects) and release updates on your own schedule. If no one steps forward, that’s fine—I’m not asking for volunteers. But if someone feels passionately that any of these are not “finished”, and has ideas for how they could be further improved, I will not stand in the way.

However, please know that I am not available for questions about the code, or for advice on how to implement changes. I’m trying to cut as cleanly as I can. Any emails I get asking about the code will likely be ignored. I’m not trying to be rude; I’m just setting expectations.

I won’t disappear, though. These libraries were just becoming millstones around my neck; without their weight dragging me down, I look forward to being able to experiment and play with new projects and new ideas. We’ll see what the future holds!

So, thanks all for a fantastic couple of years.

Net::SSH and friends, update

2009-02-02T03:58:00Z

This is a multi-project release announcement, so brace yourselves.

In short, Net::SSH 2.0.9, Net::SFTP 2.0.2, Net::SSH::Gateway 1.0.1, and Net::SSH::Multi 1.0.1 have been released. Get ‘em while they’re hot!

It’s all just bug fixes, for the most part, but if you want more nitty gritty, read on.

Net::SSH 2.0.9

The following issues have been addressed in Net::SSH 2.0.9:

If you specify a non-nil user argument to Net::SSH#start, it will override whatever setting exists for User in your .ssh/config file. (Prior to this, the .ssh/config’s User setting would always be used, regardless of what user you specified to Net::SSH::start.)
Some naughty ssh servers send channel requests even after the channel has been closed. Net::SSH handles these spurious requests better now.
Net::SSH::Connection::Session#shutdown! has been added for hard shutdown scenarios (where an error leaves the connection in an ambiguous state and you just want to terminate the connection without all that touchy-feely “graceful shutdown” nonsense).
The key loading behavior has been further tweaked so that Net::SSH will try to load public keys first (by appending ”.pub” to the given key file name), and if no public key with that name exists, the corresponding private key will be attempted. This way, operations that only need the public key will not require the (possibly encrypted) private key to be loaded in order to succeed.
The :passphrase option was being dropped on its way to the key manager. It is now passed all the way through, so that specifying the :passphrase option actually works now.

Net::SFTP 2.0.2

Only one bug fix: if an exception was raised from within the Net::SFTP::start block, it could leave the Net::SSH session trying to gracefully close channels on a borked connection. Net::SFTP will now, instead, do a hard shutdown of the ssh session connection, rather than trying to gracefully close, if an exception every bubbles up outside the block. (A side effect of this is that Net::SFTP 2.0.2 depends on Net::SSH 2.0.9, so you can’t get one without the other.)

Net::SSH::Gateway 1.0.1

Just one minor new feature was added: you can now specify the exact port number on the local host that you want the gateway to be created on. Just pass an optional third parameter to Gateway#open. This will default to the original behavior (selecting the next available port number), but now allows you greater flexibility in how you use Net::SSH::Gateway.

Net::SSH::Multi 1.0.1

Just one minor new feature: an Channel#on_open_failed callback hook was added, to mirror the on_open_failed callback hook in Net::SSH.

The future of FuzzyFinder-TextMate

2009-01-28T17:47:00Z

Back in October I released a Vim extension for mimicking TextMate’s cmd-T file lookup feature. I use it heavily now, and it works great for me.

Sadly, the author of the FuzzyFinder Vim script, upon which my extension depends, keeps changing internal implementation details that I had to hook into to make my extension work. The result? Every few weeks my extension breaks with the latest FuzzyFinder.

Needless to say, this is work I don’t need. The fuzzyfinder-textmate stuff works fine for me. It works fine for people on older versions of FuzzyFinder. And I really don’t care to support this anymore.

If you’re passionate about this, please feel free to fork the project on GitHub and release your changes independently. Feel free to post your changes on the vim script page, even! I hereby release that code into the public domain. Do with it as you please!

SQlite3-Ruby Windows Builds

2008-12-07T05:16:00Z

So, I’ve got a new sqlite3-ruby release pending (just a minor bug fix, is all), but I’ve learned my lesson about releasing a new version without a windows version pre-built.

The problem is, I’m not a Windows guy. My build environment for Windows is cobbled together and painful to use, when it works at all. I’ve decided that I won’t put myself through that pain anymore.

If you are a developer on Windows, and you have a sqlite3 build environment, and would be willing to compile the sqlite3-ruby extension for me, please do the following:

Add a comment to this post, indicating your name, and what platform you would be willing to build for (cygwin, native windows, whatever. I don’t even know the right names.)
Reload the page, and make sure no one commented before you, claiming those platforms.
Grab the latest version of sqlite3-ruby here: http://github.com/jamis/sqlite3-ruby/tree/master.
Build the extension, and email it to me at jamis at 37signals dot com.

Once I have a build of sqlite3-ruby for windows, I’ll release the new gem. (It amazes me how painful Windows makes this process. Mac OS X, Linux, BSD, etc. —you just gem install and build the source. No such luck in the Windows realm, apparently.)

"Recovering from Enterprise" video available

2008-11-29T14:40:00Z

It appears that Confreaks has posted the video of my ‘Recovering from Enterprise’ talk from RubyConf 2008. As usual, Confreaks did a great job recording the talks at RubyConf.

For those who missed it, I also posted an article version of my presentation, titled Legos, Play-doh, and Programming.

Vim Follow-up

2008-11-17T19:09:00Z

So, it’s been over a month and a half since I switched back to Vim, and I figured I’d post a bit about how things are going.

I love it. Though the future is notoriously difficult to foretell, I think it’s safe to say that I won’t be switching editors again anytime soon. Vim is where it’s at, for me.

Here’s the combination of plugins and such that I’ve found work best for me.

MacVim FTW

Any Vim will do, really, but if you’re on a Mac, you really ought to consider MacVim. Like any self-respecting Vim port, it does the command-line really well, but it also has a great OS X GUI interface.

Settings

I’ve got my <Leader> character (:h mapleader) mapped to the comma (since it’s easier to reach than the backspace character).

let mapleader = ","

I’ve installed ack and have configured Vim to use ack for grep.

12	set grepprg=ackset grepformat=%f:%l:%m

I’m a believer in spaces instead of tabs (let’s just agree not to go there), and I prefer a tab size of 2. I also like editors to try and guess the indentation level.

set tabstop=2
set smarttab
set shiftwidth=2
set autoindent
set expandtab
set backspace=start,indent

But, some file types really do require explicit tabs, and not spaces:

12	autocmd FileType make set noexpandtabautocmd FileType python set noexpandtab

(Update: apparently, python can do tabs or spaces. All my python vimming for the last 8 years has been read-only, so it never actually came up.)

I like my lines numbered, syntax highlighting on, and highlighted searches:

123	set numberset hlsearchsyntax on

I’ve got quite a few other things tweaked in my .vimrc, but those are the major biggest ones.

Plugins

FuzzyFinder

The FuzzyFinder and FuzzyFinder TextMate plugins have become essential for me. For now, installation of the TextMate-like behavior is kind of painful, but I plan to get something up on the Vim scripts index in the nearish future.

My settings for FuzzyFinder TextMate:

let g:fuzzy_ignore = "*.log"
let g:fuzzy_matching_limit = 70

map <leader>t :FuzzyFinderTextMate<CR>
map <leader>b :FuzzyFinderBuffer<CR>

NERD_tree

Thanks to all who recommended the NERD tree plugin by Marty Grenfell. It really is fantastic, definitely the best-of-breed of VIM project/directory explorers. I especially like that it is easily toggled away. I usually keep it hidden, and toggle it open only when I need to browse to something. (Thanks to the fuzzyfinder stuff, my need for a project browser is pretty small, but when I need one, NERD_tree works great.)

1	map <leader>d :execute 'NERDTreeToggle ' . getcwd()<CR>

rails.vim

The rails.vim plugin is pretty extensive, and I’m currently only scratching the surface. I don’t use snippets at all (never did in TextMate, either), but the Rake integration is pretty handy, and I’ve used the migration generator pretty often lately.

I need to spend some more time reading the docs for this one, and practicing some of the commands, since I’m sure it could turn into a real time-saver for programmers (like myself) who spend a good part of their day in Rails code.

scratch.vim

I’ve really come to love scratch.vim. Sometimes I just need to jot down some numbers, or paste the result of some query, or even take notes on a phone call. Writing any of that on a scrap of paper is a sure way to lose the info. The scratch plugin lets me take notes right where I am all day: in Vim. It doesn’t come with a way to toggle the scratch buffer (which is odd), so I wrote a quick one:

function! ToggleScratch()
  if expand('%') == g:ScratchBufferName
    quit
  else
    Sscratch
  endif
endfunction

map <leader>s :call ToggleScratch()<CR>

tComment

Maybe it’s just my own coding style, but I find myself commenting and uncommenting large blocks of code on a daily basis. The tComment plugin from Tom Link is perfect for this. You can toggle comments by using the Vim motion operators, or just do a quick visual block followed by “gc”. Good stuff!

Others

There are a few other plugins I’m experimenting with (surround.vim, vcscommand.vim), but which I use infrequently enough that they haven’t become muscle memory yet.

Workflow

Because I prefer the GUI version of MacVim, I typically have only one or two terminal windows open. I then type “mvim” to open vim from the root of whichever project(s) I’m working on, and then use ”,t” (my FuzzyFinder TextMate trigger) to search for and open files. I also use ”,b” (my FuzzyFinderBuffer trigger) to search my buffer list if I know the file is already open.

Switching between buffers with the carat character ”^” is a huge time saver. I used to use :ls (to list open buffers) and :buffer (to jump to a buffer), but the FuzzyFinderBuffer has really taken the place of both of those.

Ironically, split windows (which I missed most of all when I switched to TextMate) have taken me the longest to fit back into my workflow. TextMate trained them out of me. :) That said, I’m still trying to fit them back in, and when I remember to use them, I love them.

You?

So, that’s me. How does Vim fit in your own workflow? What settings do you prefer, and why? And what customizations are you using in your own setup?

LEGOs, Play-Doh, and Programming

2008-11-09T11:46:00Z

This article is based on a talk I gave at the 2008 RubyConf in Orlando, Florida, entitled “Recovering from Enterprise: how to embrace Ruby’s idioms and say goodbye to bad habits”.

The other day I went to Target with my son. Like most kids, I think, he’s convinced that Target is a toy store, which just happens to sell towels and shoes and cleaning supplies, too, so in his eyes it’d be criminal to not walk through the bare handful of toy aisles.

Besides, the toy section is across from the electronics section, which all geeks know is where the real toys are.

So, we went to the toy section and started browsing. I’ve always loved LEGO sets, and it’s a good thing they’re so expensive or I’d come home with a new box of bricks every time. At the Target near our home, they have half of an entire aisle devoted to boxes and boxes of LEGO sets. Need a battle-axe-wielding LEGO dwarf figure? A LEGO shark? How about a giant LEGO skull, a la Indiana Jones? And who could pass a LEGO Star Wars’ Star Destroyer model without a wistful thought or two?

It struck me at that time, though, how incredibly specific so many of these pieces are. With all of those sets in your possession, you could build a secret agent headquarters with a boulder trap that crushes angry battle-axe-wielding dwarves as they drive by in Martian exploration buggies. Which themelves are adorned with flower beds and creeper vines. And you could do all that in under 10 LEGO bricks! (Or, maybe a few more than that.)

Did you know that LEGO currently produces over 900 distinct LEGO pieces, or “elements” as they call them? Over the course of their history, there have been almost 13,000 distinct elements created. Now, that number includes variations in color and material, but even if you exclude those permutations, you’re still left with a staggering 2,800 different elements in the LEGO line.

It’s interesting that LEGO tends to encourage the use of specific pieces, rather than letting you build those pieces from more fundamental parts. It means that in order to master LEGO brick building, you have to know all of the pieces available to you, and have a good intuitive feel for how and when they should be used. That’s…a lot of information to keep tabs on. Myself, I just keep to the standard rectangular blocks and plug an exotic or two on as an afterthought when I see one that looks cool.

Also, if you’ve built up a model, and decide later that you want to change or extend some part of the model, you’ll often have to dismantle part (or all!) of it in order to do so. Kind of a pain.

Regardless, I still love building with LEGO bricks, and I suspect I always will.

Play-Doh

Now, my son being all of 6 years old, his attention span requires us to spend no more than a few minutes in any one toy aisle. So, long before I was ready to tear my eyes away from the LEGO sets, we found ourselves in the next aisle over. This was a much more colorful aisle, with bright pastels coloring various pre-school toys. My son, though, has nothing against pre-school toys, and was more than willing to drag me through them.

My eyes caught on the Play-Doh section.

The Play-Doh section at this Target is small, maybe 8 different hangers and a few square feet of shelf-space. You can get Play-Doh in as many as 50 different colors, but regardless of color, it’s all still the same thing: a bucket of malleable dough that you can pound, press, pinch, roll, and sculpt. (And rub into the carpet. And hair. And clothes. But we won’t get into that.)

Honestly, Play-Doh has a bad rap as a pre-schooler toy. It’s remarkably fun to play with. You can do all kinds of things with Play-Doh that you just can’t do with LEGO bricks. For example, the other day I built an arch out of cubes of Play-Doh that were held together only by friction. (You may not be impressed, but my 6-year-old was.)

The best part is that it doesn’t require so much memorization to become proficient in Play-Doh, though it might require more of an artistic streak than LEGO bricks do. Since I’m more engineer than artist, my Play-Doh creations tend to come out blocky and functional, rather than elegant and designed, but then, so do my LEGO creations.

Also, where LEGO models require significant work to alter or extend, Play-Doh models are dead-simple. If you want to add something to the base of your model, just graft more Play-Doh onto it. Want to change the shape of the keystone of your arch? Just pinch and mold in place. Simple!

Interestingly, I’ve found that while you can’t build with LEGO bricks using Play-Doh construction techniques, you can build with Play-Doh using LEGO construction techniques. Just build bricks out of Play-Doh. It’s unwiedly and impractical, but it can be done. The real question is: why would you want to? It’s pretty obvious that to build with Play-Doh, you should just embrace Play-Doh’s own strengths and run with it.

As obvious as that may seem, the lesson didn’t click for me for a long time. It’s not that I went about building Indiana Jones sets out of Play-Doh, one brick at a time. Rather, I didn’t realize that the same lesson applied to programming languages.

Java and LEGOs

Consider Java. Most would consider it the poster child of “enterprise” environments (though .NET is giving it a run for its money). And would you believe, Java and LEGO bricks have several things in common?

As of Java 1.6, there are well over 11,000 different classes and interfaces available to programmers in the standard library. (That’s not even counting the inner and anonymous classes that are usually not publicly documented.) Eleven. Thousand. Classes.

This is readily apparent when you consider the set of collection implementations that Java ships with.

Collection Interfaces:

Collection
Set
List
Queue
Deque
Map
SortedSet
SortedMap
NavigableSet
NavigableMap
BlockingQueue
BlockingDeque
ConcurrentMap
ConcurrentNavigableMap

General-Purpose Collection Implementations:

HashSet
TreeSet
LinkedHashSet
ArrayList
ArrayDeque
LinkedList
PriorityQueue
HashMap
TreeMap
LinkedHashMap

Special-Purpose Collection Implementations:

WeakHashMap
IdentityHashMap
CopyOnWriteArrayList
CopyOnWriteArraySet
EnumSet
EnumMap

Concurrent Collection Implementations:

ConcurrentLinkedQueue
LinkedBlockingQueue
ArrayBlockingQueue
PriorityBlockingQueue
DelayQueue
SynchronousQueue
LinkedBlockingDeque
ConcurrentHashMap
ConcurrentSkipListSet
ConcurrentSkipListMap

Abstract Collection Implementations:

AbstractCollection
AbstractSet
AbstractList
AbstractSequentialList
AbstractQueue
AbstractMap

Yes, that is FORTY-SIX different interfaces and implementations related to collections. Now, just like LEGO construction, having this volume of distinct elements on hand affects how you architect things. Writing software becomes more of a smorgasbord, where you pick and choose the specialized bricks you need, fitting them together just so. It also means that, in order to master Java, you need to have that intuitive grasp of how and when to use those thousands of classes. When do you use a HashSet versus a TreeSet? When would you use an ArrayDeque, and when would you want to subclass an AbstractQueue? It’s all part of the job.

Also, IDE’s are popular with Java in part because of the pain of refactoring. If you want to extend or modify a Java application, it can involve (like LEGO models) a lot of dismantling and reassembling.

Ruby as Play-Doh

But if Java is the LEGO of programming languages, then it could be argued that Ruby is the Play-Doh. Just as Play-Doh has been typically considered a pre-school toy, so Ruby has had a bad rap as a “toy” language, not fit for the “real world”. Also, compared to Java’s library of 11,000 classes, Ruby’s meager 1,400 classes (which number does include internal and anonymous ones, but not modules) seems paltry. And collections! Look what Ruby has to offer:

Modules:
  * Enumerable
  * Comparable (for elements within a collection)

Classes:
  * Hash
  * Array
  * Set
  * SortedSet

Just 6 options, to Java’s 46. What if you need a queue? Well, Ruby’s Array class has a queue-like interface; you could just use that. What about a sorted map? In that case, you might need to make do with a sorted set, or you could write your own, but it’s not hard. Most data structures are not rocket science, and for those that are, you can bet someone else has implemented it already.

But when you need to extend or modify your application, Ruby is a dream. Like Play-Doh, you can often just “pinch and mold” in place, grafting new code on or pulling old code out.

Ruby’s philosophy is like that of Play-Doh’s: provide a basic set of tools and make it relatively simple to build something complex with them. The very Ruby language itself is designed for this: closures, super-simple introspection of objects, runtime modification of existing objects, and the use of modules for extending classes and objects all tend to result in an environment that is simple, malleable, and extensible.

And just as you cannot use Play-Doh construction techniques with LEGO bricks, you also really cannot use Ruby programming techniques with Java. Using closures for delayed execution, or iteration, is tricky (at best) in Java, when it’s possible at all. Extending objects at runtime typically requires bytecode modification. And Ruby’s use of modules to extend classes and objects, while similar to both inheritance and interfaces, is slightly different (and arguably more powerful) than either.

You can write Ruby programs using Java programming techniques, but just as using LEGO techniques with Play-Doh is unwieldy and overcomplicated, so is mimicking Java in Ruby.

This is the lesson that I was slow to learn.

Copland

Consider exhibit A, from my Copland library.

# copland/configuration/loader.rb (collapsed)
module Copland
  module Configuration
    class Loader
      attr_reader :search_paths
      attr_reader :loaders

      def initialize( search_paths=[] )
      def add_search_path( *paths )
      def add_loader( loader )
      def load( options={} )
      def load_path( path, options )
      def use_library( name )
    end
  end
end

Copland was my first stab at a dependency injection (DI) framework, and is more-or-less a feature-for-feature port of the HiveMind project in Java. (Ironically, it was the subject of my first presentation at a Ruby conference, in 2004!)

It was designed to automatically scan directories in the load path for YAML configuration files (I’ll mention those shortly), and load them up and parse them. The thing is, I imagined a case where someone might want to use XML instead of YAML. I couldn’t just leave these folks behind! So I made the whole configuration loading framework extendible. Want XML config files? Fine! Just implement an XML parser system and register it with the configuration loader framework, and you’re good to go!

That’s just wrong on so many levels. Always, always, always build just what you need, and only when you need it. You’re in Ruby, the Play-Doh of programming languages, and the cost of adding features later is really, really low. Remember YAGNI! Obviously, this principle holds in Java, too, but it really seems like the opposite philosophy has become the standard among many Java projects. It’s too bad, because it has contributed to a bad reputation that Java probably doesn’t entirely deserve.

Here’s a classic Java pattern that just really doesn’t translate to Ruby:

# copland/class-factory.rb
module Copland
  class ClassFactory
    include Singleton

    def initialize
      @pools, @constructors = Hash.new, Hash.new
    end

    def create_pool( name, &block )
      block ||= proc { |k,*args| k.new( *args ) }
      @pools[ name ] = Hash.new
      @constructors[ name ] = block
    end

    def get_pool( name )
      pool = @pools[ name ] or raise NoSuchPoolException, name
      return pool
    end

    def register( pool_name, name, klass )
      pool = get_pool( pool_name )
      pool[ name ] = klass
    end

    def get( pool_name, name, *args )
      pool = get_pool( pool_name )
      klass = pool[ name ]
      raise NoSuchRegisteredClassException, "#{pool_name}:#{name}" unless klass
      constructor = @constructors[ pool_name ]
      return constructor.call( klass, *args )
    end
  end
end

This is an implementation of a class factory. In Ruby. The HiveMind project had a class factory, so the Copland project needed one, too!

But you know, class factories are absolutely pointless in Ruby. There are plenty of reasons for these in Java, but they just aren’t necessary in Ruby. Want a namespace? Declare the class in a module. Want the class to exist in multiple namespaces? Use constant assignment within whatever modules you desire. Need a dynamic lookup? Try #const_get. In the very worst case, just use a Hash if you need to map arbitrary strings to classes.

module A
  class B
  end 
end

# method #1, use const_get to dynamically look up classes
name = "B" 
klass = A.const_get(name)
object = klass.new

# method #2, use a hash to map arbitrary strings to classes
map = { "bimpl" => A::B }
map["bimpl"].new

Seriously. You don’t need explicit class factories in Ruby, because anything can be a class factory, implicitly.

I’ll mention one more painful Javaism that I ported to Copland. It’s so painful that I won’t even bother pasting it here—if you’re following along, look at examples/solitaire-cipher/lib/package.yml in the copland distribution.

If you do, what you’ll see are 106 lines of YAML describing how different Ruby objects in a simple, 250-line program should be initialized and connected. Yes. 106 lines of YAML. For 250 lines of Ruby.

Now, don’t get me wrong. YAML can be great for configuration. Rails, for instance, uses it for database connection information. The problem here, in Copland, was that I was using a static configuration for what would be better served with a block of code. Ruby reads elegantly; a YAML configuration file does not.

Fortunately, those wiser than myself showed me the way.

RubyConf 2004

I still remember Rich Kilmer, sitting in the front row in the October 2004 RubyConf. As I wrapped up my presentation on Copland and dependency injection, I asked if there were any questions.

Rich raised his hand. “Why didn’t you just use Ruby?”

I was confused by his question, and he had to explain. Why did I use YAML instead of just doing the configuration in Ruby code?

I think I mumbled something like “that would be a neat idea”. To me, it was a novel concept. I’d never heard of it before. You’d never see a Java program that was configured by writing Java code. That screams “hard coding”! But Ruby, you see, is different.

Ruby lets you write these beautiful little mini-languages. You’ll hear them called “Domain Specific Languages”, or DSL’s. They are subsets of the Ruby language, and you’ll find them in Rake, Capistrano, rspec, shoulda, and more. They’re really everywhere in Ruby, to varying degrees.

Although Rich tried to open my eyes, I think I would have continued to try and push Copland if it weren’t for Jim Weirich. Jim took the idea of a Ruby-ish DSL for dependency injection and made something concrete of it. A few days after the conference he forwarded me a draft of an article he was writing, in which he described dependency injection and gave a very simple (and very elegant) implementation of a DI framework in Ruby. Instead of static configuration, he’d written a basic DSL for declaring how the dependencies related to each other.

It was a moment of epiphany for me. Suddenly, I got it. I understood what DSL’s were about. I asked Jim for permission to take his simple implemention and build upon it.

The result was Needle.

Needle

Now, I’m much prouder of Needle than of Copland, because it is much closer to Ruby’s philosophy than Java. There are some pretty cool designs in there, too, though I use the term “cool” here to mean “neat without having any real practical application.”

Needle, though better, was still far from the mark.

As an example of why it misses the mark, consider Needle’s “pipeline” concept. Conceptually, it allowed you to specify a sequence of post-processors that operated on an object, allowing you to wrap code around it and mimicking (among other things) AOP-like operations. It also let me (as the author of the library) easily implement things like deferred instantiation, singleton services, and the like.

For example, suppose you wanted to declare a “deferred singleton” service, that logged all accesses to one of the methods. Underneath, Needle will create a pipeline of processors that operate on the service, returning a proxy object. The first time the proxy is accessed, it will check to see if the object has been instantiated yet. If it hasn’t, it’ll instantiate it (and cache it). The instantiation, though, actually just hands control to the next element in the pipeline, which in this case checks to see that the “singleton” constraint is enforced (e.g., all requests for this service return the same object, rather than instantiating a new object). The next pipeline element in the chain will wrap the interceptor code around the method in question, and yet another pipeline element would perform the actual object instantiation.

Pipelines really were pretty slick in Needle.

The problem, though, is that instead of leaving them as an implementation detail, I advertised them as one of Needle’s features. “Implement your own service models!” I cried. But, how often, really, is that likely to happen? Instead of exposing only the bare minimum of Needle’s API, I exposed as much of it as I could, because I could.

That’s a bad idea. Expose only what you need. The rest can be there, available, but not formally exposed. Only when (and if) you discover a need to expose more, should you expose more. This helps for several reasons.

A smaller API is easier to describe, document, and support.
A smaller API is easier for people to learn.
A smaller API is easier for you to test.
Extending a small API is much less onerous on your users than changing or restricting a larger API.

Net::SSH 1.x

Now, I’ve since come to my senses, but at one time I was completely head-over-heels in love with dependency injection. Like any schoolboy crush, it embarrasses me now to think about it, but there’s no denying it. The proof is everywhere in my project history.

Net::SSH, in particular.

At the time, I was looking for a good demonstration of the flexibility and power of dependency injection, and since Net::SSH was another of my pet projects at the time, it seemed like the perfect candidate.

I was still stuck in the “just in case” mindset, though, and Net::SSH 1.x reflected that. Badly. For instance, I isolated all the OpenSSL crypto interfaces into their own module, because “what if someone wanted to plug in a different crypto lib?” Nevermind that there was no other crypto lib for Ruby (and still isn’t, 4 years later). But WHAT IF?!?

Now, separation of concerns and modularity are good things, when used in moderation. But like any design pattern, it becomes evil when taken to extremes. Too much modularity and you wind up with component soup (and I hope you’re hungry, because you’re going to have a lot of it). With lots of tiny components, the interactions between those components can become difficult to test.

It also fuzzes the line between the public, documented API and the internal, private API. When you have two large components, it is very easy to say “A is public, and B is private”, but when you have two hundred components, where do you draw the line? It’s far too easy to let the “public” boundary meander a bit further into “private” territory than it should.

Even worse, when I added dependency injection to the mix, it became very, very difficult to follow the the flow of the program, and to understand the dependencies. Pull up Net::SSH 1.1.4, for instance, and find net/ssh.rb. Just try and figure out how a connection session is instantiated. It’s a mess. Unless you’re familiar with Needle, it’ll probably take you a long time to discover that the actual services are configured in the various services.rb files, but even after you figure that out, you still have to figure out how the different services interrelate. It’s a mess.

But, isn’t that the opposite of what DI is supposed to do? Isn’t DI supposed to improve the maintainability and testability of your code? Yeah. The problem, though, was three-fold.

First, Net::SSH, though complex in its way, was not really complex enough to need a dependency injection framework. DI itself adds complexity, and a framework for doing dependency injection adds even more, so before you go that route you need to be very sure that the trade-off in complexity is worth it. If your project is too small, you’ll actually increase the complexity of your project by adding a framework for doing DI.

Secondly, I was using a DI framework at a level that was really too granular. I was using the framework to wire together everything. No component was too small! No object too insignificant! I was on the dependency injection horse, and riding it for all it was worth. If I’d taken the time to really understand the pattern, though, I would have learned that though the pattern itself may be applied at the micro level, using a framework to do so is like nuking a mosquito—it works, but it leaves a mess behind.

Which leads to the last problem with Net::SSH’s use of Needle: it is really only appropriate for wiring together components of an application. Very, very few (Ruby) libraries will ever be complex enough, in themselves, to justify adding a dependency injection framework to them. Rather, let the application wire the libraries together as (and when) it needs to. Any more granular than that, and you’ll run into the same quagmire I did, I promise you.

Dependency Injection in Ruby

So, is there no room for DI in Ruby? There definitely is. I use DI nearly every day in Ruby, but I do not use a DI framework. Ruby itself has sufficient power to represent any day-to-day DI idioms you need. Consider this one:

class A
end

class B
  def new_client(with=A)
    with.new
  end
end

Here, B declares a factory method for generating new client objects. Because Ruby lets you declare default values for method arguments, you can let the default client implementation be A, which is the common case. But for testing, you can easily inject a mock into that method by passing an explicit parameter.

For cases where that doesn’t work, you can use a second factory method:

class A
end

class B
  def new_client
    client.new
  end

  def client
    A
  end
end

Then, in your tests, you can subclass B, overriding the client method to return your mock client implementation. It’s dependency injection, Jim, but probably not as you’ve known it.

Hashes, too, are your friend. You can allow optional arguments via hashes to specify implementation classes, defaulting to the standard implementation classes but allowing clients to inject their own implementations where needed:

class A
end

class B
  def initialize(options={})
    @client_impl = options[:client] || A
  end

  def new_client
    @client_impl.new
  end
end

“Loose coupling” and “high cohesion” are terms you’ll hear bandied about in defense of dependency injection, and those traits are certainly desirable. But strike a balance with pragmatism. There will be some who call me heretic for saying this, but don’t be afraid to introduce tighter coupling when it makes sense. Loose coupling everywhere is what I had with Net::SSH 1.x, and the result was nearly unmaintainable.

Be wise. You’re competent. Trust your instincts.

Lessons learned

If you read nothing else from this article, take to heart these bite-sized bullet-points:

Direct translations are rarely accurate. Try using the Google translator to translate a paragraph from English, to Italian, to Japanese, and back to English, and you’ll see what I mean. The same is true of programming languages. Each language has it’s own idioms, and trying to take what works well in one language and force it directly into another language is doomed to fail, more often than not.
Use your environment efficiently. Try as you might, you’ll never make a ball out of a LEGO brick by rolling it between your hands. You’ll just bloody your palms. Learn what your environment is capable of. Reading other people’s code is a great way to do this.
DSL’s, not static configuration. Ruby excels at representing DSL’s. Whenever you can, consider using a DSL instead of static configuration for your applications. You’ll find it will simplify a lot more than it complicates.
DI frameworks are unnecessary. In more rigid environments, they have value. In agile environments like Ruby, not so much. The patterns themselves may still be applicable, but beware of falling into the trap of thinking you need a special tool for everything. Ruby is Play-Doh, remember! Let’s keep it that way.
Just in time. Not just in case. Don’t play “what if” games when you’re coding. Practice discipline, and implement only what you need, when you need it. You’ll wind up with tighter, more testable code that is easier to maintain in the long run.

Learning to program is a journey, and I’m still learning, myself. I’m not perfect at applying the rules above, but I’ve found that when I do, I’m much happier. I think you will be, too.

Coming home to Vim

2008-10-10T16:16:00Z

Over three years ago, I was faced with a dilemma. I had recently switched to the Mac (from Linux) and was still using my text editor of choice (vim), but at the time, vim’s “integration” with OS X was pretty minimal (and that’s putting it optimistically). I experimented with emacs, but it never clicked for me, and honestly, emacs on OS X wasn’t all that better than vim at the time. Sadly, reluctantly, I said good-bye to vim and switched to TextMate.

TextMate was (and certainly still is) a fantastic text editor. The project drawer was awesome, finding files via cmd-T was super powerful, and smarter auto-completion and snippets promised a new and faster way to pound code. After a couple of months of reteaching my fingers how to edit text, I was happy.

Sometimes, though, late at night, I would think again of vim.

Fast forward three years. The vim landscape is different now. There is actually a Mac-friendly GUI version of vim now, MacVim, which actually looks like it belongs on OS X. Vim 7 supports UI tabs, and a much more powerful auto-completion mechanism than before. And plugins like rails.vim and fuzzyfinder.vim mean that TextMate no longer has a corner on powerful project navigation.

For the last few weeks I’ve been toying with switching back to vim. TextMate’s “snippet” feature never clicked for me, and the only times I used it were by accident (when it annoyed me more than it helped me), but I really was hooked on the project browser, and cmd-T, and a few other things. I realized that, with a little work, perhaps a way could be found to reimplement most of the things I loved about TextMate, in vim.

This last week I’ve worked exclusively in vim, to test that theory. It’s like coming home. As I said, TextMate is a powerful and wonderful editor, too, but differently powerful and wonderful. Vim’s wonderfulness and power is the wonderfulness and power of git, or linux, where the learning curve is steep (ridiculously steep at times), but the rewards of mastery are sublime. I didn’t even realize I had missed a sane shift-J, or using the dot key to repeat the last command. Fix transposition typos with ‘xp’. Select a single word with ‘viw’. Drop bookmarks with ‘m’, and jump right back to them with single quote.

If any of that makes your stomach roil, then vim is not for you. :) But to me, it’s like being embraced by a long-lost friend after years apart. And vim holds no grudges.

There was still the issue of the TextMate features that I had particularly come to love. First to tackle was cmd-T, since my workflow had become so dependent on that for finding files. Takeshi NISHIDA’s fuzzyfinder.vim script seemed like exactly what I wanted…at first. It’s definitely a powerful tool, but the fuzzy finder for files was not TextMate’s cmd-T, and my instinctive attempts to treat it so were causing me a lot of aggravation.

So I took an evening and wrote fuzzy_file_finder, a Ruby library that mimics (and improves on, if I do say so myself) TextMate’s cmd-T functionality. Then, I extended fuzzyfinder.vim with fuzzyfinder_textmate, which bound the fuzzy_file_finder to vim. The result?

See for yourself: http://s3.amazonaws.com/buckblog/videos/fuzzyfinder_textmate.mov (600K, QuickTime video).

I’m still working on a solution for the project browser. Yes, I know there are several (“countless” might be a more accurate term) vim plugins that present a project drawer in a split window, but even before being spoiled by TextMate those didn’t feel right to me. I’m experimenting with a cocoa tree view that sends files to a specific vim server, and it mostly works, but I’m still not sure it’s the right solution. If I do come up with something, I’ll definitely open it up and share it. (On the other hand, if any of you out there in readerland already know of such a thing, please point me at it!)

So, I’m still reacquaiting myself with all my old muscle memories, but here are some commands I wasn’t previously very familiar with which are proving useful in conquering my TextMate habits:

:e [file]. This is great if you don’t want to use fuzzy finding. Vim will even do tab completion to make things easier.
:ls. Shows all of your buffers.
:buffer [file]. Fantastic for quickly switching between buffers. You can give it just part of a file name and it will jump to the buffer that matches.
The ctrl-6 (technically ctrl-caret) key is awesome for switching back and forth between two buffers. For instance, if I just need to quickly look at one buffer, I can jump right back to where I was with ctrl-caret.
ctrl-W introduces a whole host of options for working with split windows.

And lastly, can I just say that Vim is seriously the poster-child for documentation? I recommend spending 15-30 minutes, every day, in :help, just exploring. There is a LOT there, and all excellently documented.

So, all you vimsters out there: what commands do you frequently use? What features of vim are you so dependent on that you’d be useless without them? Do share!

Capistrano 2.5.0

2008-08-29T03:11:00Z

Capistrano 2.5.0 is now available! You can read the full release announcement on the capify.org news blog.

SQLite3-Ruby 1.2.3

2008-08-27T15:11:00Z

SQLite3-Ruby version 1.2.3 is now available. It is a maintenance release, fixing just a few things:

The permissions on the database.rb and translator.rb files in 1.2.2 were incorrect, resulting in broken sqlite3-ruby installations for many *nix users. This is now fixed.
A few more Ruby 1.9 compatibility issues were patched.
Some optimizations were applied to speed up iterating over result sets.

To install or upgrade:

  gem install sqlite3-ruby

Thanks!

Cap 1.4.1? Go 1.4.2. Now.

2008-08-26T15:43:00Z

Are you currently using Capistrano 1.4.1? If so, drop everything (I mean it, do this RIGHT NOW) and install Capistrano 1.4.2.

Why, you ask?

Capistrano 1.4.1 will work just fine, right up until you decide you want to experiment with Capistrano 2. When you do that, Cap 2.3+ will install net-ssh 2.x, which kills Capistrano 1.4.1 in all kinds of really obscure ways.

The good news is that Cap 1.4.2 is completely compatible with Cap 1.4.1. It adds no new features. The only “bug” it fixes is that if you ever install net-ssh 2.x, Cap 1.4.2 will still happily continue to work.

Ultimately, an upgrade to Cap 2 is recommended, but I understand it’s not feasible for everyone. So, if you’re one of those who can’t go cap2 yet, please please please PLEASE PLEASE FOR THE LOVE OF ALL THAT IS HOLY upgrade to Cap 1.4.2. It’ll make your life easier, and it’ll make my life easier (because I won’t have to keep troubleshooting the same issues over and over). Thanks!