I am a big fan of John Gruber’s website Daring Fireball. He always has good information and I appreciate his opinions on technical issues. (And yes, I have been supporting him for years, I have a whole collection of t-shirts.). With the whole brouhaha surrounding AT&T’s new data plan pricing, I have been getting increasingly frustrated with all the moaning and bitching people have been doing lately. Gruber has been covering it and agreeing with some of it, so as I was getting fed up this morning, I penned a long email to him. As I don’t want him to be the only recipient of my wisdom, here it is.

John,

I’ve been getting increasing frustrated with the whining I am hearing all over the net regarding the changes to the AT&T fees for data. I am hardly an AT&T fanboy, but this has just gotten ridiculous, I think people are just looking for anything to bitch about.

There seem to be two issues at stake here: data costs and tethering.

Data Costs:

For anyone who uses < 2GB month, this is a better deal by at least $5/month. Simple, no questions asked. If you always use < 200MB a month, you save $15/month. Also simple.

</p>

If you occasionally go over those limits, it is still probably a good idea. In my case (and I would have thought I was a power user, but I guess I find WiFi more often than not), I looked at the last 7 months and I went over 200MB once. So, I would have to had paid an extra $15 that month, but saved $15/month the other 6. Still a good deal.

As for that tiny percentage who uses > 2GB/month consistently, ok, fine, they can bitch, but I want pictures of their data usage.

Tethering:

Yes, AT&T is charging people $20 for no more data usage, BUT, people forget that AT&T is a business therefore they are not setting their $25/month price to cover the cost of every user using 2GB/month. If they did that it would be much more expensive. They use an average value.

This is where the extra cost from tethering comes from. I believe it is safe to assume that an iPhone user without tethering will use less data on average than a user with tethering. AT&T believes they can make a profit by charging handset users $25/month based on “average” consumption and tethering users $45/month because their average usage will be higher.

Is any of this “fair?” Business isn’t about being fair, it is about making money. And that is what AT&T is doing. If someone doesn’t like it, switch carriers. But if they are going to keep their iPhone and AT&T, I wish they would stop complaining.

Thanks for being the recipient of my minor rant. As always, I love your coverage.

-Jeremy

When we last left our intrepid hero (geek), had convinced myself that the new Western Digital drives with the 4kb block sizes operated perfectly fine under OpenSolaris, but that the frankenstein of my old server needed to be replaced. I figured if I was going to do this, I was going to do it right. And given the goal of this machine was storage, which meant fast network, lots of SATA ports, and lots of drive bays. So, thanks to the wonders of the internet and Newegg, I found the following items.

• CPU: Intel Core 2 Duo E8400 Wolfdale 3.0GHz - Seemed a nice balance between power consumption, power, and performance.
• Motherboard: TYAN S5220AG2NR Toledo q35T LGA 775 Intel Q35 ATX Motherboard - This won out due to the low cost and large number of on-board SATA ports (6).
• Memory: 4 x Kingston 1GB 240-Pin DDR2 SDRAM DDR2 667 - I bought ECC RAM, but unfortunately the motherboard does not support it (descriptions were all vague).
• SAS Controller: LSI Internal SATA/SAS SAS3081E-R 3Gb/s PCI-Express 1.1 RAID Controller Card - This card is one generation back for SAS, but it is plenty fast to support 8 SATA drives. I didn’t get this until after the machine was up and running, I will touch on moving the array to the new card later.
• Storage: 8-2 (aka 6) x Western Digital Caviar Green WD15EARS 1.5TB 64MB Cache SATA 3.0Gb/s 3.5” Internal Hard Drive - 2 drives were DOA on arrival, but Newegg did a great job of handling the RMAs. Chosen because they are big, relatively cheap, low power, and cool to run.
• Power: Antec CP-850 850W Power Supply - Specifically designed for the case below, lots of power connectors, and power to spare.
• Case: Antec Twelve Hundred Black Steel ATX Full Tower Computer Case - This is a beast of a case, but lots of room inside and nice to work with.

If you look at the specs and are counting, you will see that this gives me a combined total of 14 SATA ports, more than enough for expansion as the case has 12 drive bays. The primary storage is the 6 new SATA drives attached to the LSI SAS card. I also brought over the 3 remaining 1GB SATA drives I had from my old server attached to the motherboard SATA controller. I don’t trust them completely anymore, but they make a nice big scratch space volume.

Of note is also the challenge I had with getting enough working drives. I needed 6 hard drives to build the system, but I had to order a total of 8 before I had 6 working ones. I’m not sure what to take away from this. Obviously, I don’t like the idea of drives dying, but I think I much prefer DOA drives than ones that will fail later. But these are obviously consumer grade devices, which is why they are going into a dual-parity RAID system because I don’t trust them.

In my old server, I used an old ATA drive as the boot drive. It was small (capacity) and loud and probably used up more power than it should. This time, I wanted something small for the boot drive, since I didn’t really care about it terribly much. So, for the new server, I took a 2.5” laptop SATA drive and plugged in a SATA->USB adapter and am using that as the boot drive. It doesn’t take up much room and draws very little power.

So, after all the parts arrived and 2 rounds of RMAs with dead hard drives, I built the new system. Obviously OS installation was the first step. So I temporarily hooked up a DVD drive into the system and booted it up. BIOS, check. Bootloader, check. Kernel loaded, check. Detecting devices………….nothing. Shit.

I ended up doing quite a few steps at this point of debugging, trying older and newer versions of OpenSolaris. No change. I finally learned how to boot the kernel such that it drops into the kernel debugger (if that doesn’t make you run in fear, it should) and get it to boot while spitting out lots more debug info. The specific commands for anyone coming to this page with similar troubles were to first add the parameter -kdv to the kernel line in GRUB, then when you get dropped into the debugger, enter the following items.

use_mp/W0
moddebug/W80000000
::cont

This hardly gave me a culprit. But I noticed after enabling and disabling various things in the BIOS that the problem always seemed to occur around the time the system loaded the ATA drivers. As the only thing attached via ATA was the DVD drive, I researched in that arena. (My apologies that I don’t have all the links I used to find the info here, but I was more concerned with getting it working than preserving the link history for posterity, but you do get my solutions.) In the end, I found other people who reported that their CD/DVD drives did not function properly with DMA enabled. I added the following option to the kernel boot:

-B atapi-cd-dma-enabled=0

And, voila! The system booted into the installer. To be honest, this would not work as a long term solution, not using DMA for the DVD drive makes it slow and bogs down the rest of the system, but since the only use of the drive was for install, I didn’t care.

The install of the base OS went smoothly as did the upgrade to the latest development release. While in general I will say the userland tools for OpenSolaris are lacking as compared to Linux and even that the package tools are lacking in general, upgrading the whole system is a breeze.

pkg install SUNWipkg
pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org
pkg image-update

Next came the CIFS (Windows file sharing) server.

pkg install SUNWsmbskr
pkg install SUNWsmbs
svcadm enable -r smb/server

Now on to building the new drive arrays. Given that I never make things easy on myself, I still had a slight conundrum. One of the 6 new drives that was destined for the new machine was still in the old server as the replacement for the drive that died. So that meant that I only had 5 drives for the new machine where I wanted 6. AHA! but this was going to be a dual parity RAID system (raidz2 to be specific). This means that once operational, it can sustain a 2 drive failure without loosing data. The question arises, can you create an array from scratch that is in a degraded state and then add the drive later? The answer is yes! The basic idea is to create a sparse file to be a temporary “drive” for array creation. Then immediately remove it from the array (degrading the array) since it won’t actually work to store data on it since there wouldn’t be enough drive space on the boot drive to hold all the parity. Then, once the data is transfered, “replace” the fake drive with a real one and let the system heal itself.

# Create temporary "drive" so I don't loose
# redundancy on source array
mkfile -nv 1500g /TEMP-DRIVE
zpool create -f nas raidz2 c9t0d0 c9t1d0 c9t2d0 c9t3d0 c9t4d0 /TEMP-DRIVE
zpool offline nas /TEMP-DRIVE
# Do all of the transfers and install 6th drive
zpool replace nas /TEMP-DRIVE c9t5d0

All good! Now on to transferring the data from the old system to the new. One of the (many) awesome features of ZFS is the ability to snapshot filesystems. On top of that, you can easily serialize these snapshots and send them around and then have them unserialized on another filesystem or even another machine. A great feature is that I could be transferring data to my new server without having to stop using my old one for the bulk of the time. Again, the basic structure was to:

1. snapshot the old system and transfer the data over to the new server (while still being able to use the old server). This took about 20 hours, but as I had no loss of usage, that is ok.
2. Stop using old server, take another snapshot, and transfer it to the new server. The delta between the first snapshot and the second is small, so the transfer time was also small (less than an hour).
3. Power down the old server, start using the new server

And here are the gory details:

zfs snapshot -r nas@transfer-primary
zfs send -v -R nas/scratch@transfer-primary '
    ssh gothmog@rivendell.home pfexec /usr/sbin/zfs recv -v -u nas/scratch
zfs send -v -R nas/av@transfer-primary '
    ssh gothmog@rivendell.home pfexec /usr/sbin/zfs recv -v -u nas/av
zfs send -v -R nas/backup@transfer-primary '
    ssh gothmog@rivendell.home pfexec /usr/sbin/zfs recv -v -u nas/backup

# Stop all use of system on bywater

zfs snapshot -r nas@transfer-final
zfs send -v -R -i nas/scratch@transfer-primary nas/scratch@transfer-final  '
    ssh gothmog@rivendell.home pfexec /usr/sbin/zfs recv -v -u nas/scratch
zfs send -v -R -i nas/av@transfer-primary nas/av@transfer-final '
    ssh gothmog@rivendell.home pfexec /usr/sbin/zfs recv -v -u nas/av
zfs send -v -R -i nas/backup@transfer-primary nas/backup@transfer-final  '
    ssh gothmog@rivendell.home pfexec /usr/sbin/zfs recv -v -u nas/backup

Honestly, not terribly gory for what I did. Note that transferring that way brought along everything about the filesystem, including the NFS exports, the CIFS exports, permissions, ACLs, everything. Very painless.

At this point I was largely done. But as I mentioned in the hardware list, I didn’t get the SAS card until later. I realized that it was a waste to not use the 3 old 1TB drives from my old server. Given their age and the failure of one of them, I didn’t trust them lots, but using them to move my network scratch space off the raidz2 array to a separate striped array seemed like a good use. As I had no extra SATA motherboard ports, I needed an external card. The LSI cards had been recommended to me, so I picked one up. It is very well supported by OpenSolaris. I changed the firmware on the card to the IT firmware (that is the non-RAID firmware since I was not going to use the hardware RAID) which took a little work to get a DOS boot environment, but that is another entry.

As it is the better card, I wanted the raidz2 array on the new card. Again, ZFS to the rescue. Under Linux this would have been a pain, but it was extremely simple under OpenSolaris.

# prepare the array to be moved
zpool export nas
# power off box, swap cables to new card, power up
zpool import nas

That was it.

Since then, I have installed a few more things to get things working well.

• Smartmon to monitor the disk drives. (OpenSolaris article)
• Apcupsd to shut down the system on power failures
• Virtualbox to host a few test environments

Simply put, I am thrilled with the new machine. I expect it will last me for quite a while. Oh, the Barad-dûr reference? You will have to wait for the pictures.

A week or so ago, in my weekly status email from my OpenSolaris NAS, I got an unfortunate notice.

status: One or more devices has experienced
   an unrecoverable error.  An attempt was made
   to correct the error.  Applications are unaffected.

The last sentence is the welcome news that one gets from using cool things like ZFS. But, the first sentence strikes fear in the heart of someone who has nearly 3 Terabytes of data stored in that drive array. So, it was clearly time to replace the failing drive and get the array back up to full redundant goodness. The simple solution would have been just to replace the aging drive and be done with it. But since I am at about 90% capacity on the array, I figured I would take the opportunity to grow the size of the array by replacing not just the single failed 1 terabyte drive with a new one, but purchase 4 new 1.5 terabyte drives. I know there are 2 terabyte drives out there, but they are significantly more expensive and I didn’t want to go there.

Here is where I started losing the battle which later ensued. I was traveling in Boston at the time and decided to order the drives while I was on the road so that they would arrive soon after I returned and I could get things moving faster. I settled on the Western Digital Caviar Green WD15EARS 1.5TB 64MB Cache SATA 3.0Gb/s 3.5” Internal Hard Drive from Newegg. I liked the low power consumption, the large size, and the cool running. However, I did not do my homework on Western Digitals new “Advanced Format” feature.

This feature changes the atomic storage unit on the drive from 512 bytes to 4 kilobytes. In the abstract this is good. It means that the drive itself is more efficient and needs less overhead on the platter to store error correction information (ECC) resulting in more storage space. However, it means that if you ever try to write out less than 4 kilobytes of data to the drive, it needs to read the entire 4 kilobytes from the drive platter, “merge” in your new data, recalculate the ECC, and write it back out. Again, doesn’t sound bad. But if your operating system assumes that the disk block size is 512 bytes, it always writes in chunks that are aligned on 512 byte boundaries, not 4 kilobyte boundaries, resulting in TONS of read-modify-write cycles, absolutely killing throughput.

Guess what assumption OpenSolaris makes? If you guessed 4 kilobytes, you are wrong, and don’t know Murphy too well. 512 bytes it is. So, while the drive will work, it’s performance will absolutely suck. I mean bad. The system was estimating it would take 4.5 DAYS to move about 800 gigabytes of data onto the new drive. And while 800 gigabytes is a lot of data, it should not take that long.

So, turns out I was wrong, Solaris does just fine with the drive, it was just my assumption of speed that was wrong. I will not go into the day long gnashing of teeth that occurred as I tried different things and tested various drives on the machine. In the end, I convinced myself that the drive was operating acceptably and that it would in fact take 3 days or so to do the resilvering.

But, my efforts also brought out the kindness of the internet and old friends. I posted a frustrated tweet and a good friend of mine from college, Monique, emailed me saying that her husband was an expert in Solaris, ZFS, and hard drives and asking if they could help. Eric and I emailed back and forth for much of the day, me sharing what I had done and thoughts I had about the problem, and he helping me with ideas and questions. I can’t thank them enough and I certainly owe each of them a few beers (or other beverage of choice) next time I see them.

But, in all of this, I realized the frankenstein machine I have built for this data is, well, a frankenstein of a machine. And as this has become my central data store at home, I decided to engage in some retail therapy and have purchased the parts for a new, better computer to house the system. This one will have room for 9 drives, if I ever get up to that point, and a new motherboard/CPU/power supply to more comfortably handle the system. It will no longer be shoehorned into a Dell case that was not designed to hold that many drives. So, I am looking forward to building my new system when all the parts arrive later this week (hopefully). If I think of it, I will take pictures and post a montage.

Off again. This makes the fourth week in a row I am traveling. Back down to Jacksonville, FL on my way to Lake City, FL. For those of you keeping track, yes, I was in Florida two days ago. The necesity of this trip seems to be receding even as I travel, but such is life.

But rather than drag on about the negatives, I will talk about the great day I had yesterday. Coming back from Florida, Tiffany planned a great day on Sunday. It was probably the first day this year where being outside was comfortable. We got together with Adam, Liz, and their new puppy, Casey and went first to the New Kent Winery. We were served our tasting by Al, who did a great job. Good stories and good information on the wines. Wine was good enough that I bought a few bottles to restock my “cellar” at home.

We then drove into Richmond to Brown’s island and had a great picnic of bread, cheese, prosciotto, and apples. Sitting on the grass in the sun could not have been better. Liz, Adam, and Tiffany all practiced their gymnastics on the lawn to my amusement. We wandered around the island for a bit and made our way on the rocks in the James for fun. Casey also seemed to have a great time playing on the rocks as we walked.

Our last stop was James River cellars which was a bit if a dissapointment compared to New Kent. The servers were not as knowledgable and the wines were not as good.

We headed back home at that point with Casey mostly passing out in the back seat between Adamand I. After dropping off Adam, Liz, and the puppy we enjoyed a nice dinner of Top’s China and Dollhouse. Wish I had a longer weekend but I could not have asked for a better day between weeks of travel.

Looking forward to a quick week and a fun weekend to come.

No, this will not be an entry about parapalegic hexapods. I am on the middle set of flights this week. Monday I made way up to Green Bay, WI to help make a sales presentation to a potential client I have been involved with for a month so. This was the same client which required the trip to Iowa and Illinois two weeks ago. Now I am on the second leg down to Jacksonville, FL for two days of internal meetings including a tour of one of the first Target sites I worked on.

The presentation was an interesting one for me. I have s new respect for my friends who have done their PhD defenses. We had 4 hours to present and go through a gauntlet of questions and critiques of our proposal. While I have little experience in the sales arena, I think it went well. I should know something when we land.

Unrelated to the meetings, one of the clients recomended what turned out to be an amazing restaurant, the Republic Chophouse in Green Bay. Beyond having the Plungerhead Zin, which is a great wine, I had a great bit of Wisconsin cheeses, followed by a suprisingly good salad. I like salads, but this had an amazing basil vinagrette that must have had tons of garlic. It had a wonderful basil flavor with a sharp, raw garlic bite too it. For the main course, I had a Morroccon lamb chops that were unbelievable. Wonderfully prepared medium rare and with such depth if flavor. Cinnamon and other spices melded so well wih the meat. Delicious. An unexpected surprise in Green Bay.

So hopefully Florida is ok. Then back home Saturday. Cya!

So, I am here again writing in the air coming back from San Francisco trying to admire the view of the Rockies. While it truly is an amazing site, it doesn’t quite compensate for the travel. I set an internal goal to try to write an entry once a week, but given that I hit Friday without a positive, interesting topic coming to mind, I am left only to be slightly grumpy about travel. I hope you will grant me some leeway or just skip this entry.

Last week I did a whirlwind tour of Iowa and Illinois in 2 days. This week involved 3 days start to finish in California visiting the main bottling facility for E&J Gallo wineries. While not the idilyic setting of a winery with grapes stretching off into the distance it was cool to see a facility that bottles more wine than probably anywhere else in the country. I have never seen so much wine before.

Next week I had been scheduled to spend Wednesday night through Saturday night outside of Jacksonville, FL for an offsite meeting. But, as is often is the case, plans rarely survive first contact with the enemy or the customer. Thursday I was asked to add a pretrip Monday and Tuesday to Green Bay, WI. So, my current plan involves 6 flights next week starting at 6 am Monday morning taking me through the following airports: RIC->ORD->GRB->ORD->JAX->IAD->RIC. This over 6 days. Ugh.

So, not in the best mood right now. Hopefully I can have some quality time with Tiffany between the laundry and stuff this weekend.

There you have it. A grumpy blog post. Deal.

(But, in an attempt to mitigate the grumpy, you get a series of pictures. Aren’t I considerate?)

Trucking along at flight level 390 sitting in 2B on a United Airbus A320 right now. Ahh. What a great beginning for a blog entry. The warm nuts (just don’t say anything) and a glass of wine make this as pleasurable as air travel can be for me. Which means that it is only slightly better than the prospect of getting the same root canal my boss is getting tomorrow.

This trip started yesterday, which happenned to also be my two year anniversary with Swisslog. Up to now, I have concentrated on our largest customer on already sold projects. While I have helped on proposals before, my role has been behind the scenes. Behind the scenes no longer!

We took a prospective client to two of our sites, one in Iowa and a second in Illinois. I like to think that I availed myself well. While I am proud of the trait, my rather strict belief in openess and honesty has a tendancy to make sales guys just a little nervous. I have never never been able to embrace the loose grasp of reality necessary to truly be a salesman.

But even so, I always feel just a little dirty after sales meetings. Thankfully, I think I availed myself well and I believe that we can provide this client with a very good solution, so the sales effort was easy.

So, back home tonight after a short but tiring trip. Next week? Another sales trip to Modesto, CA. Hopefully I get some good wine out of it.

Twitter being the source of all good things nowadays (at least according to “social media experts”), I posted a question to my huge list of followers for ideas to write a blog entry about. Parsing through all the suggestion (1), I found the best idea from KT:

tisfan: @gothmog talk about how you got into photography?

Good ideas coming from KT are not unexpected as she is a far more prolific writer than I am with her blog, The Hungry Little Caterpillar and other non-web-enabled writings. So, I will embrace her question and try to do my best.

My first memory of photography was taking the occasional picture with my father’s SLR when I was very young. I don’t remember any of the pictures I took (there may not even have been film in the camera). I do have rather distinct memories of the mechanism of the camera. The manual film advance lever; the wonder of what lenses could do; the split screen focusing through the viewfinder. As a budding geek, the pure mechanics provided wonder to me that made the act of taking photographs irrelevant. My father had a box of glass lenses (why, I don’t know) that I would play with for hours exploring the various properties of convex and concave lenses. And by “properties” I do not mean solar destruction of insects; I got a fresnel lens later for that.

At some point before I was less than 10, I got a 110 film camera. I really remember nothing of the camera or any pictures I took with it, but I do remember the weird film cartridges that it used. It was a very very basic camera with manual advance and no adjustments, just point and click. I think it might have used the “single use” flash cubes, but I could be wrong.

When I was about 10 years old, I got my first 35mm camera. It was a Canon Snappy S, and it was red. I thought it was the best thing ever. I found some photos I took with it including ones that I took on a school trip to Quebec City (the waterfall to the left are the Montmorency Falls). I still have the camera; it is in my closet. It was a fully automatic camera, but it had a cool rotating lens cover that I was enamored of. I don’t remember spending much energy taking pictures at the time; I have only a few rolls of developed film.

While I did take a few photos during high school (mostly while I was at governor’s school), I stopped through college. I don’t really know why, there were just so many other things going on; I lost interest. My reintroduction to photography came in step with the digital camera revolution. But, as I look back over those images there is an inkling of my later photography topics with lots of nature shots; waterfalls, rainbows, and sunsets.

In 2001, I purchased my first digital camera, a Canon Powershot Digital Elph S110 (Canon has a way with names). I still have it and use it today especially given that I have a waterproof case for it. There are pictures in my cruise album from snorkeling and river tubing taken with it. It was, and is, a great camera. At about the same time as I bought the camera, Elizabeth and I planted lots of flowers in the front yard (photos). These flowers became subject for my budding photography.

This camera went on many a trip with me; my first cruise, Washington DC, Ocean City, Atlanta, GA, and many more, but in the end, I started getting frustrated with the limited capabilities of a fully automatic camera. I think the difficulty in taking macro photographs bothered me the most. It was not the fault of my little camera, but without having control over focus, things just got hard. I remember trying to take pictures by holding my hand where I wanted the focus point to be and then moving it to take the shot. Not a recipe for great photos.

So, on July 23rd, 2006 I ordered my lovely DSLR, a Canon EOS 20D with 17-85mm f/4-5.6 IS USM lens (yes, I found the Amazon order confirmation email). I’m pretty sure I can credit Tiffany for getting interested in having an SLR as she talked quite a lot about her photography classes in college. I’ll also freely admit that it was a cool, geek toy. At that time, having a DSLR was not very common and I liked having a new thing. But once I started using it, I fell in love with it. I remember one of my first photo trips was wandering around campus and I saw a bunny near Swem Library. I was able to track it with the camera and using the rapid shot get several frames of it running away. That was something I could never have done with my old camera.

While there have certainly been ebbs and flows in my photography, it is an art form that I really love. Part of that love comes from the fact that I am reasonably good at it. It has allowed me to capture things which are purely artistic, but I am also able to capture and save the memory of other activities I love, whether it be cooking, travel, or my friends. It is also wonderful to have someone I love who shares my interest. Tiffany and I can reliably spend hours wandering around a park, botanical garden, or Colonial Williamsburg boring anyone who is with us as we investigate a cool shadow or reflection. So I think that is the true answer to KT’s question. I started and continue having a passion for photography because it helps me capture and save the things I love in the world.

Most of you know (or maybe not) that I have a bit of an obsession with the Supreme Court.

As the only of the three branches of government that I really have any respect for, I enjoy reading and listening to the workings and decision making process of the justices. To be clear, respect does not always mean I agree with the decisions, but even when I don’t, I fully respect the members of the court and the institution as operating in a way I like. Contrast that to the other two branches of government where I sometimes agree and sometimes disagree with the outcomes, but I rarely respect the process and motivations of the members.

Right after the State of the Union, I was poking around iTunes to see if anyone had posted the audio (to no avail), but came across a set of interviews that C-SPAN did with each of the Justices. They are available as podcasts. Each runs between 30 minutes and an hour and it is just a one-on-one interview with the justice. So far, I have listened to Justice Alito, Justice Scalia, Justice Sotomayor, Justice Kennedy, and former Justice O’Connor. Each is unique as you would expect, but I have loved them all. While I have had the pleasure of seeing Justice O’Connor, Justice Scalia, and Justice Ginsburg speak in person, it is not often that most people hear the Justices speak publicly; this is a great opportunity if you are curious.

If you are interested in more resources, I also recommend American Original: The Life and Constitution of Supreme Court Justice Antonin Scalia which I read on our cruise last month. A few other books I have enjoyed on the court are Becoming Justice Blackmun: Harry Blackmun’s Supreme Court Journey, The Majesty of the Law: Reflections of a Supreme Court Justice by former Justice O’Connor, The Supreme Court by former Chief Justice Rehnquist, and The Brethren: Inside the Supreme Court.

They (the infamous they) say that the first step in dealing with a problem is to admit that you have one. So, this entry.

“My God!” you must be thinking, “what problem is this that Jeremy feels the need to announce on his blog for all the world to see?” It must be horrible; I might be traumatized; Maybe I should get a hanky, just in case. Could it have something to do with the bursty nature of his blog posts? (hat tip, Marcia.) So, here we go, deep breath.

I have become a food snob.

::GASP::

Yes, I know, horrifying. What does this mean? How can I live with it? Can I only eat expensive food produced by pompous chefs wearing funny hats? Must I speak with a bad french accent? No. But it does mean I want my food to be, put simply, good. With very specific definitions of good. I’ve developed a set of guidelines or maybe inequalities that I try to follow:

• Local is better than trucked in from half way around the world
• Things I could (in theory) grow/produce myself is better than something that requires a chemistry set. (Not sure how Wylie Dufresne fits in here)
• Single instance restaurants are better than chains
• Passion for food matters
• Presentation matters
• Taste matters
• Outstanding service matters
• Respect your food, it is more than just what gets shoved into your pie whole

So, what brought me to this point? I blame many things. First of all, experiencing food by great chefs who also live by those standards: Chef Everett at The Blue Talon, Chef Power at The Fat Canary, Chef Kennedy at Dudley’s Farmhouse Grill in Williamsburg; Mas Tapas in Charlottesville; Craft in New York City; Marmalade in San Juan, Puerto Rico. Visiting great local restaurants everywhere: Pierce’s Pit BBQ here in town, The Shake Shack in New York City, cubbyhole pizza places in Florence. Food television and books (aka food porn) have certainly contributed; Alton Brown, Anthony Bourdain, Iron Chef, Top Chef. Tiffany certainly gets some credit too; it is invaluable to have someone who shares the same passions with you. It would be very hard to do this by myself, and would look more like an addiction than a healthy passion.

What does it mean for daily life? Chain restaurants are basically out, certainly for dinner. I try to buy “real” foods as much a possible. I savor food for foods sake when I eat; I try to taste food the same way I taste wine. When travel opportunities present themselves, I almost always start by searching for great local restaurants if the destination is fixed, or I look for a great new restaurant I want to try and use that to pick a destination. Restaurants with tasting menus get first billing. What is that I hear? Tasting menu? You don’t know what that is? Oh, sweet goodness.

Tasting menus are the chef’s equivalent of saying “mine is bigger than his.” They are multicourse meals (6-17, yes 17) where the chef and kitchen shows you, the diner, why they are the shit. Each course is rarely larger than a single bite (maybe two) that have been created to be a perfect bite of flavor, texture, presentation, smell, everything. Very often paired per course by just the right wine, it is, well, about the most decadent way to experience food. You almost always get very attentive service with explanations of each item and ingredient and why they should go perfectly with each other and the chosen wine. Often times the chef will come out and say hi and thank you. I can explain it in no other way than it is an amazing experience.

It filters into cooking at home as well. I don’t do it nearly as often I should, but when I do, I try to pay special attention to the food I buy, the food I cook, how I prepare it. I don’t have to be fancy in my food, but I try to put the same love and caring into it that I want when I eat a chef-prepared meal. And on occasion, cooking something fancy is just plain fun.

So, there you have it. I’m a food snob. I have a “problem” and I love it. It is a problem I would wish on everyone I know; you will enjoy life more. (And probably be healthier for it.)

Now if only I could could get a gig as a travel food photographer…