One day around Christmas 2008, my dad came across a box of old slides and films; that night we set up our antiquated projectors and had ourselves a good old fashioned slideshow. I believe (or at least we think) many of these photos were taken by his father, A. Herbert Campbell, some time in the 40′s and 50′s. (For the sake of our family history, I should probably revisit these and record the actual dates and people instead of just guessing.)

I took the lo-fi route and captured a few of my favorite projected photos with my DSLR while the slideshow was ongoing. There are certainly better ways to preserve old photos, but I really like how these turned out. This one featured three of my favorite things: a cat, an Adirondack chair, and a tree, all neatly arranged into thirds:

This one was badly washed out, but I loved the ethereal quality so much that I came back to San Francisco and attempted to create a painting based on the photograph. (The result wasn’t good enough to include here, but one can find it over on the blog that I share with Candice.) I believe that’s my grandmother, Helen, on the left, with A. Herbert to the right:

Here’s my Uncle Tom. I never got to meet him, but from all the stories he seemed like a swell guy:

I think my Aunt Lupie (Helen to the rest of the world) identified herself in this one. I love the wagon and the barn door and the blue sky and the headscarf and just about everything:

Mmm, mmmilk. And an interesting angle here:

Another ghostly visage, unidentified:

Likewise, here. I think I just found my next album cover:

I spend a lot of time idly looking out our windows, watching the fog and clouds that drift past. Life in the Inner Sunset is often dictated by the low clouds that peel back for (at best) a couple hours in mid-afternoon before roaring back towards downtown just before supper time. I got a bit lucky this weekend in that I was able to capture the three most common variations on this theme: 1) dense, quickly moving fog; 2) thick, slower moving clouds that periodically tease us with views of the downtown skyline; 3) light cloud cover that reveals blue skies before engulfing the entire city just before sunset. I was slightly disappointed that the blue sky lasted so long on Monday if only because it makes for a less interesting time lapse, but the dark clouds that take over at the very end are worth the wait. At the very least, I’m glad that I was able to capture a typical summer scene before our perfect Indian summer sets in this week.

Anyway, the video is available on YouTube and embedded below as well (best viewed at 720p). The frames were captured using my Canon 7D. I created the video clips using Lightroom’s slideshow mode, but unfortunately all I had available for editing purposes was an ancient version of iMovie, so the quality is awful, especially during the fade transitions. I learned a lot this weekend, and next time around I’ll use more care in the way I shoot (and post-process).

There have been the usual wallet-draining equipment upgrades every few years. A cheap point-and-shoot film camera in 1996. My first digital camera [a Kodak DC 280] in 1999. A better point-and-shoot digital [a Canon S60] in 2004. My first DSLR [a Canon XTi] in 2006. And finally, the leap to a Canon 7D earlier this year.

There were different organization methods and tools… Originally I just scanned photos in and scattered them haphazardly on my hard drive. Later I tried some shoddy photo manager from Ulead, which was so limiting (only 256 characters for comments? binary file format?) that I started writing my own photo organization software in 1999 (called JAlbum, not to be confused with a later, presumably more successful product of the same name). I rarely used the desktop app part of JAlbum, but there was a barebones photo album server component to it that I ran on my own Linux box for a few years. Eventually I got tired of managing (and hearing) my homebuilt server and photo software, so around 2005 I decided to give in and just get a Mac mini for serving photos to the few friends that wanted to see them, and started entrusting my expanding photo library to iPhoto.

When I upgraded to a Flickr Pro account back in 2006, I attempted to draw a line in the sand, thinking that I would only post new (presumably only good) photos on Flickr. I’d keep all the old photos backed up through some combination of CD-Rs and DVDs and hard drives strategically placed around the country. When I started shooting RAW full time in 2007 and moved on to Lightroom, I made a similar decision (or so I thought) to just keep the new/good photos organized in Lightroom, and leave the rest sitting in iPhoto for posterity.

The reason I mention all this boring history is to make it clear that my decisions never last, and I’m always thinking that there must be a better way to shoot photos and to organize them. Specifically, I’ve been worried about losing old photos, even if only a small handful will be of interest decades from now. So a few weeks ago, as if I had tons of spare time on my hands (which I no longer do; my days are filled with chasing the little tyke around the house), I got it in my head that I would REALLY get my money’s worth from my Flickr Pro account and use it as a backup-backup solution for all my photos.

The general plan was to write some scripts to migrate all my 1997-2008 photos over to Lightroom, preserving as much of the keywording, titles, and captions that I had added over the years in iPhoto and my homebrewed apps. Then once I was satisfied with that, I would find any of the pre-2006 photos (and videos) not already at Flickr and make sure they were available there as well. When complete, I wanted to make sure that for every photo in my Lightroom catalog there would be a corresponding item at Flickr, and vice versa (no duplicates, all organized into folders/sets, no loose ends).

1. Migrate iPhoto albums to Lightroom folders

Prior to using iPhoto, I organized my photos on my hard drive by placing photos for a given day in a folder named “MMDD (Single Event Title)”, like this:

    Pictures/
             2003/
                  ...
             2004/
                  0307 (Alamere Falls)
                  0501 (Coachella)
                  0515 (Mendocino)
                  0617 (Mean Toledo)
                  0703 (At Camp)
                  0704 (At Camp)
                  0708 (Adirondack Chair)
                  ...
             2005/
                  ...

When I started using iPhoto, I preserved this directory structure, but I also organized them into iPhoto albums, which allowed for logical groupings of multi-day events and made it easier to upload a whole album to a Flickr set. At the time, I preferred to have most recent albums at the top of the album browser, so the above folder structure might have translated to iPhoto albums like this:

    Y2005
       ...
    Y2004
       July 4th in Red Hook
       Mean Toledo
       Mendocino
       Coachella
       Alamere Falls

However, once I started using Lightroom a few years ago, I settled back on a naming convention that was a hybrid of those two approaches, placing all photos from a logical event into a folder named “MM.DD Event Title”. (In case of multi-day events, MM.DD refers to the first day of the event.) So, I decided to write a little AppleScript that would take a selection of iPhoto albums and copy each album’s contents into a folder using this new naming scheme, for example:

    Pictures/
             2003/
                  ...
             2004/
                  03.07 Alamere Falls
                  05.01 Coachella
                  05.15 Mendocino
                  06.17 Mean Toledo
                  07.03 July 4th in Red Hook
                  ...
             2005/
                  ...

In addition, I wanted to preserve any keywords, titles, and captions that I had added in iPhoto over the years. I also wanted to include the original directory name somewhere (since in some cases it provided a little more detail than the album name did, especially for multi-day albums). For this, I turned to Phil Harvey’s awesome ExifTool utility. In my AppleScript, I extracted the iPhoto keywords, titles, and captions, added the original folder name to the caption, and fed this into ExifTool for each migrated photo. Later, when importing the photos into Lightroom, this data would be pulled automatically from the IPTC tags in the JPEG files and included in the Lightroom catalog.

You can find the resulting “Export iPhoto Albums” AppleScript here (MIT licensed). It didn’t take too long to put together. It was a little tricky making sure that titles and captions that included single or double quotes were escaped properly before calling out to exiftool via the shell (always a nightmare when mixing shell-script-like syntax with another scripting language). Also, I ran into a bizarre issue that I had managed to forget from years ago, when Candice and I each brought cameras to Greece, and through some crazy coincidence both CF cards were taking photos in the same filename range (e.g. we both had photos named IMG_1234.JPG), so when my script ran, some of my photos would overwrite hers, or vice versa. It took me a while to figure out why the file count was different between the iPhoto album and the exported folder. I didn’t take the time to fix it properly, so you’ll see a gross hack in the code. Just something to be aware of if anyone else tries to use this script.

2. Import photos (and videos) into Lightroom

Once I had all my iPhoto albums exported to folders for each year from 1997 through 2007, importing them into my Lightroom catalog as folders was a snap. However, I wanted to use this opportunity to finally organize all the little video clips I had taken over the years with my S60 and Nexus One. Prior to Lightroom 3, my only option was to keep videos roughly organized (well, in most cases, completely unorganized) in some folder on my hard drive. But now with Lightroom 3, it’s possible to keep videos organized alongside photos in folders and collections, so I took some time to put old videos into their corresponding folders.

It’s worth mentioning that the S60 saved videos as AVI files, which Lightroom 3 handles just fine, but my Android phone saves videos with a 3GP extension. After some googling, I found that 3GP is a variant on MP4, which Lightroom 3 does support, so luckily it was just a matter of renaming those video files to have a .mp4 extension instead of a .3gp one, and Lightroom imported them without any trouble.

(On the other hand, Lightroom doesn’t seem to recognize the original capture time of these videos. I spent a couple fruitless hours trying to figure out a way to create sidecars for all the video files using ExifTool to extract the capture time so it would appear in the Lightroom catalog metadata, but I never got it to work so I just gave up. In most cases, the capture time isn’t critical, since it can be inferred from the file’s timestamp or from just guessing based on adjacent photos, so no big loss.)

3. Export photos (and videos) from Lightroom to Flickr

Here’s where it got pleasantly easy. I’ve been using Jeffrey Friedel’s fantastic “Export to Flickr” Lightroom Plugin for exactly as long as I’ve used Lightroom. It makes exporting from Lightroom to Flickr very simple, but during this recent project I discovered two features of the plugin that make it even more of a gem.

When it came time to upload my pre-2006 photos and videos to Flickr, I wanted to continue my usual process of exporting each Lightroom folder to a corresponding photo set on Flickr, but I wasn’t looking forward to tediously uploading one folder at a time, entering in the name of the Flickr set each time, over and over again (a hundred or so times). I was delighted to discover that Jeffrey’s plugin has support for “automatic destinations”, where you can automatically specify the destination Flickr set using metadata from the photos that are being uploaded. (See the “add/edit auto destinations” link in the “Upload Destination” section of the “jf Flickr” export dialog in Lightroom.) There are tons of built-in templates and tokens, so, for example, you can use the photo date or folder name as the name of the set. But in my case, I wanted a way to convert my Lightroom “MM.DD Event” naming convention to my less nerdy, date-prefix-less Flickr “Event” naming convention. It turns out that Jeffrey’s plugin is based in part on Lua, so you can put a Lua expression (i.e., code) in there for more powerful templating. A quick Google search revealed the Lua syntax for getting a substring, so I could convert “MM.DD Event” to “Event” with the following one-liner:

    {LUA=string.sub(Folder, 7)}

Now, it was possible to select a whole year’s worth of photos, click the Export button, and by the next morning they would all be up on Flickr, neatly organized into sets according to their folder names. So much better than the manual approach.

4. Link existing photos (and videos) on Flickr to items in Lightroom

My usual workflow in Lightroom is based around a Smart Collection called “Unfiled”, which includes all photos that don’t have the “Uploaded to Flickr” flag set. (I consider a photo as “done” once I’ve edited it and uploaded it to Flickr, so this shows me all the ones that need work.) But what about all those photos that I had uploaded to Flickr between 2006 and 2008 before I started using Jeffrey’s plugin? I wouldn’t want to re-upload them to Flickr, and I was getting worried that I’d have to go through all those Flickr photos and manually link them to my Lightroom catalog so that they don’t show up in my “Unfiled” collection.

That’s when I discovered a second, awesome, previously-unknown-to-me feature of Jeffrey’s plugin, which is buried up in the File / Plugin Extras / Flickr Extras menu. Basically, you can select a batch of photos, then navigate to that menu item and select “Associate Images Automatically”, and the plugin will match photo dates from your Lightroom catalog to existing photos on Flickr in an attempt to figure out if they’re the same, and if so, it will mark them as “Uploaded to Flickr” and save that Flickr URL in your Lightroom catalog. This worked about 99% of the time; the only troublesome photos are ones taken in burst mode or ones that have virtual copies (like if I uploaded an image in both color and black-and-white), but these are easily resolved using the “Associate Images Manually” button. These features were a huge time saver, so three cheers for Jeffrey!

5. Find stray photos that aren’t properly linked between Lightroom and Flickr

After completing these four steps (a couple weeks later), I was certain that the end was in sight and that I would soon resume normal life, but alas, Flickr reported 10,547 items, and my Lightroom catalog claimed 10,529 items. Since there’s no way I could rest soundly knowing that I was a miserable failure of a human, with a complete lack of symmetry in my photo collection and therefore my life (overstatement, yes, but this is how my brain works), I decided to forego sleep and wrote a Perl script (yes, Perl; someday I’ll learn something more hip) that would fetch all the photo page URLs from my Flickr account, and would compare that list to the ones recorded by Jeffrey’s plugin in Lightroom.

The first part was kind of a pain. After a few minutes of fruitless googling, I concluded that there weren’t any programs out there that would simply fetch your photo page URLs from a Flickr account. (There are plenty that will download the photos themselves, but not the URLs.) I had used some Java libraries years ago to interact with Flickr’s APIs, but lately at work I’ve found myself to be pretty handy with curl and simple scripts, so I wanted something quick and dirty. I looked into the various Flickr libraries for Python, Ruby, Perl, and even PHP (yow) but they all seemed to require actual thought and some amount of bootstrapping work. That said, I went with the Flickr::API library for Perl and found a few brief but confusing examples on the net showing how to use them. Eventually (after first signing up for a Flickr API key) I wrangled together some Perl code that would download all of my photo identifiers, from which I could figure out the corresponding photo page URLs.

Up until then I had assumed that it would be easy to export metadata (such as the Flickr URL for each photo as recorded by Jeffrey’s plugin) from Lightroom to, say, a simple text file. It turns out that there is no built-in support for metadata export, but fortunately another fellow named Timothy Armes wrote a similarly awesome Lightroom plugin called LR/Transporter, which allows for exporting metadata summaries to a plain text file. To capture a list of all the Flickr URLs from my Lightroom catalog, I selected File / Plugin Extras / Export Metadata using LR/Transporter, then Summary File, and in the “Specify text to add for each photo” field I specified:

    {pluginField[info.regex.lightroom.export.flickr2,url]}{return}

Feeding the resulting text file into my diff-flickr-lr.pl script (also MIT licensed) produced a simple HTML page (with photos and links) revealing a couple dozen discrepancies between my Lightroom catalog and Flickr collection, some of which were simple mistakes on my part when using the “Associate Images Manually” approach mentioned earlier, some of which were caused by photos and screenshots I had uploaded to Flickr years ago but never maintained in iPhoto, and a few of which turned out to be Candice’s paintings that were stored in my Lightroom catalog but uploaded to her own Flickr account instead of mine.

Finally (finally!) after clearing up those issues, I checked my Flickr account: 10,565 items. Then, my Lightroom catalog: 10,565 items. Now I can sleep soundly.

Sadly, the sidewalks at that intersection were replaced (and the misspelling corrected) years ago, and I never took the time to take a photo before that happened. Fortunately, the city is full of similarly jumbled sidewalk etchings, and now many years later I find myself walking around with a camera-enabled phone in my pocket, so I finally started to document these misspellings. Herewith a small sampling of those that I’ve found thus far.


Sometimes two letters get swapped…

SHOTEWLL [sic]
18th St and Shotwell St

Sometimes the wrong letters get doubled…

CAAP ST [sic]
18th St and Capp St

Sometimes they use an upside-down M for a W…

WEBSTER [upside down M]
Post St and Webster St

Sometimes they use an upside-down W for an M…

MARIPOSA [upside down W]
Mariposa St and Harrison St

Sometimes a letter gets flipped the wrong way:

FLORIDA [backwards D]
18th St and Florida St

Sometimes they realize their mistake and try it both ways…

F0NSTON [upside down U on right side up U]
Funston Ave and Kirkham St

Sometimes they use a number as a substitute for a letter…

HARR1SON ST [sic]
17th St and Harrison St

Sometimes they get the abbreviation wrong…

19TH TS [sic]
19th St and South Van Ness Ave

Sometimes they forget a letter…

KIRKAM [sic]
28th Ave and Kirkham St

Sometimes they add one letter too many…

OFFARRELL [sic]
O’Farrell St and St Joseph’s Ave

Bonus! It’s not a street name, but this is my favorite sidewalk etching in San Francisco. Yes, it’s a bit crude, but I think it’s great that of all the things this person could immortalize in concrete, they chose this visual pun…

TWO MEN WALKING A BREAST
9th Ave and Irving St

My compulsion is unlikely to go away any time soon, so I’ll continue to keep this Flickr set updated as I find more. If you know of any particularly good ones, add a comment and I’ll go check it out.

The hardest part of this change is leaving my great colleagues at Sun, many of whom I worked with since my earliest days there as a quiet co-op student (intern) and later as a new hire. I’ve learned so much about software development, and a great number of other random/wacky topics as well, from this team. They are a unique bunch, replete with humor, brains, generosity, egos, caring, quirks, ingenuity, and great storytelling abilities. They’re like a family, and that’s why I’ll miss them.

So, what’s next for me? On Monday I will be starting at Plausible Labs, a worker-owned software cooperative here in San Francisco. Java developers may know its founder, Landon Fuller, from his work on SoyLatte and porting OpenJDK to Mac OS X. I’m really excited to be working with Landon and Terri. It should give me a chance to learn new technologies (iPhone and Android, among others) and to see what life is like outside a corporate monolith. I like the concept of the co-op (a relatively rare setup in the software world) and I’m hoping we can make it a viable way to run a business. I also like the idea of building local, community-focused software — creating things that I would use myself, rather than building for unknown “others.”

(I’m also very happy to be working closer to home; no more Caltrain! Well, really I’m just trading one long commute for a shorter one involving the equally unreliable Muni, but at least now biking to work is a reasonable option.)

As part of this transition, I’ve finally set up this new website and blog, where I can keep my professional and personal activities organized in one location. I’m not planning to post to my java.net blog any longer (since I expect to be blogging about non-Java related topics in addition to the usual Java stuff here), so if you’re accustomed to reading my (increasingly infrequent) entries there, be sure to update your bookmarks.

I’ll stay subscribed to a few of the usual Java and OpenJDK-related mailing lists, so I’m not planning to disappear completely. That’s the great part about OpenJDK: it’s possible for me to remain involved even though I’ll no longer be with Sun. (I wish I could say the same for JavaFX!)

Happy 2010! It should be a fun year…