May Contain Blueberries

the sometimes journal of Jeremy Beker


Shortly after I started my new job at Kolide in March, I was presented with a very interesting problem. We dynamically create views in our database to perform certain queries that are specifically scoped within our system. These views are then queried by our webapp before being destroyed when they are no longer needed. This had been working perfectly fine since the original feature was created, but due to the fact that the view needs to be created, all of the work had to be performed on our primary (writer) database. This presented several problems:

  • It added load to our primary database for what was functionally a read-only operation
  • It required that we do some machinations to switch users to ensure that the view was restricted as much as possible
  • There was a danger that an out of control query could impact our primary, writer database

My project was to come up with a way to do the same operations, but use our read only replica to perform the queries themselves to isolate them as much as possible.

The high level solution was pretty straightforward:

  • Create the view on the primary
  • Execute the query using the view on the read-only replica
  • Remove the view on the primary

A simple implementation would look something like this in ruby:

ActiveRecord::Base.connected_to(role: :writing) do
  create_view_function
end

ActiveRecord::Base.connected_to(role: :reading) do
  execute_query_using_view
end

ActiveRecord::Base.connected_to(role: :writing) do
  delete_view_function
end

However, if you run this code, you will fail on execute_query_using_view with an error stating that the view does not exist. And this is where things get complicated.

Tangent on Postgres replicas: When you have Postgres databases set up in a primary-replica relationship the replica is kept up to date by receiving updates from the primary as changes are made to the primary database. This is done through the WAL (write ahead log). Before the primary database writes out data tables to disk, the complete set of transactions are recorded in the WAL. This allows the primary to recover the database to a consistent state in the event of a crash. But it also facilitates replication. The replica streams the contents of the WAL from the primary and applies those changes to its data tables resulting in an eventually consistent copy of the primary. And therein lies the rub, the replica is potentially always a little behind the primary.

So the error we received that the view did not exist on the replica is really that the view does not exist on the replica yet. The replica has not had enough time to replay the WAL and get the view in place before we tried to query against it.

This requires a more refined solution then:

  • Create the view on the primary
  • Wait until the view has been replicated
  • Execute the query using the view on the read-only replica
  • Remove the view on the primary

Tangent on transaction IDs: Every transaction that is committed to the database is assigned a transaction ID. Per the documentation “The internal transaction ID type xid is 32 bits wide and wraps around every 4 billion transactions.” There are useful functions that can be used to query the status of the transactions that have been committed to the databases.

So, in order to make this work, we have defined two functions that can be used to help understand what has been committed and where.

#
# Return the txn id of what has actually been written out from the server's internal buffers
#
def committed_wal_location(connection)
  # Get the current writer WAL location
  results = connection.exec_query("select pg_current_wal_lsn()::text")

  # the result is a single row with a single column, pull that out
  results.rows.first.first
end

and:

#
# Checks to see if the connection passed has replayed the txn to its tables
#
def has_replica_committed_wal_location?(connection, wal_location)
  # create bind variable
  binds = [ ActiveRecord::Relation::QueryAttribute.new(
      "id", wal_location, ActiveRecord::Type::String.new
  )]

  # Get the current writer WAL location
  results = connection.exec_query("select pg_wal_lsn_diff ( pg_last_wal_replay_lsn(), $1 )", 'sql', binds)

  # the result is a single row with a single column, pull that out and convert to a number
  # negative means that this connection is behind the WAL LSN that was sent in (lag)
  # zero means we are in sync
  # positive means the replica is in the future, shouldn't happen :)
  !results.rows.first.first.to_i.negative?
end

Using these methods, we can build a more robust system that accomplishes our goal:

ActiveRecord::Base.connected_to(role: :writing) do
  # Check out a connection to the pool and return it once done
  ActiveRecord::Base.connection_pool.with_connection do |connection|
    begin
      # Generate dynamic views
      connection.exec_query_with_types(create_statements)

      # Explicitly commit this so that it gets pushed to the replica
      connection.commit_db_transaction

      # Get the WAL current position so that we can wait until the replica has
      # caught up before executing the query over there
      view_creation_timestamp = committed_wal_location(connection)

      # Run the query on the replica
      ActiveRecord::Base.connected_to(role: :reading) do
        ActiveRecord::Base.connection_pool.with_connection do |ro_connection|
          # Verify that the replica is at least up to date with the primary before executing query
          retries = 0
          while !has_replica_committed_wal_location?(ro_connection, view_creation_timestamp)
            raise ReplicaLagError.new("Replica has not committed data in #{retries} tries.") if retries > MAX_REPLICA_RETRIES
            retries +=1

            # Exponential backoff to wait for replica to catch up
            sleep(0.1 * retries**2)
          end

          results = ro_connection.exec_query_with_types(view_sql)
        end
      end
    ensure
      # Drop the views that we created at the beginning
      connection.exec_query_with_types(drop_statements)
    end
  end
end

With this code, we accomplished our goal of being able to create views dynamically, use views to restrict data access, but still gain the advantages and protections of actually executing queries against the views on our replica systems. As of this writing, this has been in production for several months without any issues.


I love tracking data. This can be in the form of computer and network tracking using Zabbix, personal health data through my Apple Watch and Apple Health, or, as is the subject of this post, weather data. But sometimes getting things working can be really frustrating and I wanted to document what I have found setting up my most recent acquisitions from Netatmo so that someone else can hopefully benefit from it.

I have had a Netatmo weather station for a few years now and have been quite happy with it. I recently acquired another indoor station as well as a rain gauge and wind gauge. Setting those up is where I ran into some issues but finally solved. I received the additional indoor unit first and setting it up using the iOS application went as expected and things started working just fine. I only started having issues when the outdoor rain and wind sensors came.

I tried setting them up using the iOS application and they would never connect to the main unit. No matter how many times, it just failed. Upon searching the Netatmo support site, I found a reference to a Mac application that could be used to add modules. While very finicky and repeatedly telling me it had failed to add the modules, it in fact did.

Unfortunately while the rain and wind sensors worked, the outdoor and additional indoor module now failed to work. So I removed the two units and used the iOS app to add them back. But now the rain gauge and wind sensor failed.

After much trial and error I removed all of the modules and then re-added them all using the Mac application. This is what finally got them all operational at the same time. My theory is that the two methods are subtly different and overwrite data for the other.

So if you have issues (or honestly even if you don’t) I would recommend using the Mac (or Windows) application to add all Netatmo modules.

Now back to enjoying all my data.


I always enjoy seeing how others have set up their work environments but I realized that I have never shared my own. I’ll do my best to give a list of all the items in the picture as well as what I have tried to achieve.

Philosophically I want a very clean workspace. I find that if my surroundings are neat and clean that I am able to focus more on whatever I am doing whether work or play. This applies particularly to what is in my eyeline as I am at my desk. The rest of the room is generally neat, but if I can’t see it directly, I worry less about it. More specifically this means not having clutter on my desk especially when it comes to wires. My goal is to have as close to zero wires visible from where I sit. I am happy that I have achieved that as much as I think possible at this point.

The Desk

The desktop itself is part of an old desk that was given to me by a dear friend when I got my first apartment over 20 years ago. It is oak veneer over particleboard, I think, but it is very solid and very heavy. I converted it years ago to have metal legs which is why you can see the intentionally exposed ends of carriage bolts at the corners. Those are no longer used but I still like the look. About a year ago, I replaced the legs with adjustable legs from Uplift. They are not the cheapest but the quality is excellent and the support from Uplift is great. I had one leg that started making an odd noise and they sent me a replacement no questions asked.

Yes, there are many computing devices on my desk. Here is the rundown.

Computers

  • minas-tirith - Apple MacBook Pro (16-inch, 2023) with an M2 Pro - This is my work computer in a Twelve South BookArc partially behind the right hand monitor.
  • hobbiton - Apple MacBook Pro (16-inch, 2021) with an M1 Pro - This is my personal computer.
  • minas-ithil - Mac Pro (Late 2013) with the 3 GHz 8-Core Intel Xeon E5 - I always wanted to have one of these and I recently came into one. It is used to run a few applications that I want always on. It also looks pretty on my desk.
  • sting - iPhone 14 Pro on a Twelve South Forté. Currently running the iOS 17 beta so I can use Standby Mode to show pictures.
  • narsil - 12.9” iPad Pro (5th generation) with an M1 and [Apple Pencil]

Computer Accessories

  • 2 LG 4K Monitors (27UK850). They are fine, nothing special about them.
  • Amazon Basics dual VESA mount arm. Nothing special here either, it works. I don’t move the monitors around any, so it does an acceptable job holding them in place.
  • Caldigit TS4 - This is the center of my system. It is a single Thunderbolt cable to my work MBP and all accessories are plugged into it. Not cheap, but rock solid and solves for my “no visible cables” policy. It connects to both monitors, all the USB accessories, and wired GbE to my home network.
  • Caldigit TS3+ - I had this before upgrading to the TS4. I still use it to attach my personal MBP to things.
  • Apple Magic Keyboard with Touch ID and Numeric Keypad - I like the dark look and the Touch ID is great for authenticating especially for 1Password.
  • Apple Magic Trackpad - I have been known to switch back and forth from a trackpad to a mouse, but I am in trackpad land now.
  • Dell Webcam WB7022 - I got this for free at some point and I mostly love it because it reminds me of the old Apple iSight camera.
  • Harmon/Kardon Soundsticks - These are the OG version of these from around 2000 that I got with an old G4 Powermac. They still sounds great.
  • audio-technica [ATR2100x-USB] microphone - Got this on the recommendation of the fine folks at Six Colors.
  • Elgato low profile Wave Mic Arm - I had a boom arm before but I hated having it in front of me. This does a great job of keeping the mic low and out of camera when I am using it.
  • Bose QC35 II noise cancelling headphones - I don’t use them at home all that often but they are amazing for travel
  • FiiO A1 Digital Amplifier - This is attached to the TS4 audio output and I use it to play music to a pair of Polk bookshelf speakers out of frame.
  • Cables - I generally go for either Anker or Monoprice cables where I do need them. I like the various braided cables they offer.

Desk Accessories

  • Grovemade Wool Felt Desk Pad - In the winter, my desk gets cold which makes my arms cold. This adds a nice layer that looks nice and absorbs sounds as well.
  • Wood cable holder - I didn’t get this exact one but I can’t find where I originally ordered it from. The idea is correct in that it allows you to have charging cables handy but not visible. Really a game changer for having cables nearby but out of sight.
  • Field Notes notebooks - I like to keep a physical record of what I work on. There are no better small notebooks than these.
  • Studio Neat Mark One pen - I am not a pen geek, but this one is just gorgeous and I love writing with it.
  • Zwilling Sorrento Double Wall glass - This is great for cold beverages in humid climates as it won’t sweat all over your desk.

Lighting

  • 2 Nanoleaf Smart Bulbs - These are mounted behind my monitors facing the angled wall to provide indirect light

Underneath

I am actually pretty proud of the contained chaos that is under the desk. It is kept in place enough that I don’t hit cables with my legs which is more than I can say for many of my desks in the past.


Last Friday in my 1:1 with my boss, I came in quite frustrated as a result of a code change that I had to roll back that morning. He rightly observed a few facts. 1) I have only been at my new job for 2 months, so I am still new to the code so I needed to cut myself some slack, but more importantly 2) he has observed that I feel that I have failed if something does not work well the first time I deploy it. While related to 1, it speaks to a good observation about myself and also another subtle learning I have to make about my new role.

The less interesting point is that I am a perfectionist and I need to work on being more forgiving with myself. Noted.

The more interesting point I think is to expand on his statement “I expect that my code will work properly when I release it the first time.” I thought about that a lot over the weekend and I actually think there is an addition to that statement that explains my frustration better. “I expect that my code will work properly when I release it if I expect my code to work properly.” This is a weird circular statement; let me explain.

When I work on a code change, I have an internal sense of how I want the code to work, how risky I think the change I am making is, and the areas where I think there could be problems. A lot of this is intuition. And I do not propose to release my code until I have reached a certain comfort level that I understand the “shape” of the code and its impact on the system when it is released. The frustration came from my intuition of the risk of what I was releasing being way off base.

This intuition will only come with time so for now all I can do is be kinder to myself until it gets more accurate.

(Apologies for the bad title. For whatever reason I couldn’t get past mixing Great Expectations with The Wrong Trousers.)


The last time I went through a job transition, I was moving from a role in technical management to being an individual contributor again. This involved picking up a new programming language, a new framework, and a new industry. I knew that I would be starting from scratch in a lot of ways; hoping my experience would carry me through.

This new transition is markedly different. I have left a role as a very senior engineer where I had significantly more professional experience than my coworkers as well as significantly longer tenure at the company than all of my coworkers (by nearly 5 years). I exaggerate but it could be argued that I knew everything about everything of our technical stack and how we had gotten to where we were. My new role is using the same language and the same framework (but in a much more modern way).

Intellectually I knew that this would still be a big change, but I don’t think I appreciated how much of a shock it would be going to basically knowing nothing about anything. This has been the hardest adjustment to make. Coming to every problem with only basic knowledge; like knowing a language but never having spoken to a single person for whom it is their native tongue. My new coworkers have been amazing and are spending lots of time helping but this is a process that can’t be forced.

In my head, I imaging a vast map of all the knowledge I could know that is covered up like the map of an RPG. As I experience and learn, I open up different parts of the map, but they are islands. Over time I will start making the connections between different parts and that is when I truly can bring my experiences to bear and add more value to the team.

It has only been a few weeks, but I feel like some small connections are being made. I can’t wait until I get more. It is an exciting and sometimes exhausting journey.


Just a quick entry that will hopefully help someone else with a similar situation. (Pardon the ton of links and buzzwords, I want to make sure this is easily searchable.) In my network, I use the ISC DHCP server via OPNsense. Most of my hosts get fully dynamic addresses and I have the DHCP server register their names in DNS using RFC 2136. This works very well in that I don’t have to worry about manual IP allocation yet I can still use friendly hostnames to access systems. For the few systems that need staticly defined IP addresses, I have set up static leases via system MAC addresses. This has worked very well so far.

But I recently came across a new situation that hit a snag. As I will now be having 2 laptops that I use regularly, I wanted to be able to attach them to my desktop monitors via my CalDigit TS3+ which has a wired ethernet connection. This means that I will have two computers that will at times be requesting an IP address from the same MAC address. This isn’t an issue particularly except that macOS will set the hostname of the system based on a reverse DNS lookup of the IP address it receives. Given caching and the timing of the RFC 2136 updates, I would open a terminal on laptop minas-tirith and see my prompt saying my computer was hobbiton. This bothered me for aesthetic and possible confusion reasons.

DHCP has the ability to send a “client ID” when requesting an address and the ISC DHCPd server can do a static IP assignment based on that client ID. This seemed the perfect solution. I could set a client ID on each of the laptops for that network interface and each would get the proper IP, register a DNS name for that IP and all would be happy. My first attempt almost worked. I set the client IDs on both the laptops and the DHCP server did give them different IP addresses so I was confident that the client ID was being used, however, they were not the static assignments that I had set in my configuration.

Doing some searching I found very little about this problem but I saw a few mentions that indicated that DHCP clients often prepended some bytes to what they send as the client ID. I dug a bit into what was being sent by looking at the dhcpd.leases file on my server and lo and behold, that was what was happening:

lease 192.168.42.208 {
  starts 4 2023/03/02 12:08:46;
  ends 4 2023/03/02 12:10:05;
  tstp 4 2023/03/02 12:10:05;
  cltt 4 2023/03/02 12:08:46;
  binding state free;
  hardware ethernet 64:4b:f0:13:22:f6;
  uid "\000hobbiton";
}

The uid line is the client ID. The client was prepending a null byte at the beginning. So, I went back to my DHCP server and set the matching client ID name to be \000hobbiton. I renewed the lease and VOILA! I was now getting the IP address I assigned.

Another step in living the dual laptop life. Now if I can just find a good solution to using Apple bluetooth keyboards with TouchID…


As I mentally prepare for my new role and the new computer that will come along with it, it seemed like a good time to do some digital housekeeping. At Food52 I never had a company owned laptop so I was able to be a little more lazy about keeping work and personal things separate. But a new shiny M2 MacBook Pro showed up a few hours ago and I want to try to do things a bit cleaner now. In addition to that I wanted to improve some security and identity items.

Overall the setup went pretty well taking only a few hours. I’m sure I missed some things, but I am ready to get started!

SSH Keys

For the longest time I was a bit inconsistent with SSH keys. I wandered between them representing my person as an identity and having them represent me as a user on a particular computer. With the advent of being able to store and use SSH keys via 1Password, I wanted to clean things up. Using 1Password, it made more sense to treat keys as something that represented me personally without regards to the computer I am on. I reverted to having 2 keys stored in 1Password, a big (4096 bit) RSA one and a newer ED25519 one. I prefer the newer key but I have found that some systems can’t handle them so having both is nice. I cleaned up my access to various SSH based system and now have a simple authorized_hosts file with just 2 keys in it everywhere. (GitHub just gets the ED25519 one as they support it just fine.)

API Keys

I don’t have a lot of API keys right now but I assume I will have more in the new role. Another new 1Password feature (can you tell I am a fan) is command line integration for API keys. I had read about this when I was at Food52 but had not gotten around to setting it up. I did so this morning for a few keys I still have and it works really well. Excited to see how it works when I have a bunch more.

Software

You may already know about brew for installing UNIX-y software. But did you know about using a Brewfile? You can use them to install all kinds of applications automatically with one command. This simplified the vast majority of installs on the new laptop.

File Synchronization

The convenience of tools like Dropbox and iCloud Drive is pretty obvious. But for one like me who is very concerned with privacy (and likes futzing with tech and occasionally making things more difficult for myself), I don’t like the idea of keeping my sensitive data on someone else’s infrastructure in an unencrypted format. So, a number of years ago I started using Resilio Sync (at the time it was BTSync). This is a sync product that operates in a similar way to Dropbox but it is peer-to-peer between any number of computers you control. It also has the ability to set up read-only and (more interestingly) encrypted copies. This means I can have a replicated server that has all of my data but it is inaccessible to anyone who breaches that machine. This has allowed me to set up a few remote servers outside of my house that provide disaster recovery but are also safe from a privacy perspective.

As part of my cleanup, I made a new shared folder specifically for work files separate from my personal synced folders.


At least the titles work together now.

I have spent more of my work life hiring new engineers than I have spent looking for work myself. One of the observations I have made when looking at hiring is that there is a high drop off at each stage of the interviewing process. So, you can’t think of just how many you need to hire, you need to think of how large your pipeline is to support the number of roles you want to hire.

Obviously every company and situation is different but I expect a 10x dropoff at each major stage:

  • 100 people apply
  • 10 people participate in interviews
  • 1 person is hired

This could probably further be broken down when you include initial phone screens and multiple rounds of interviews but this rubric has served me well when hiring. So, how has this worked out from the other side, when looking for work?

  • 25 applications
  • 5 (meaningful) interview processes
  • 2 late stage interviews
  • 1 job offer (accepted)

So these numbers are not the same as the other side, but I think interesting in that it was a factor of 5 dropoff at each stage.

Oh, yeah, I guess I buried the lede, I got a new job. The team was great to talk with, I like the company, and the work they do, so I am super excited. They were quite understanding that I need some time off to clear my head of my last job and the immediate job search so I don’t start for a few weeks. I will share more later.


I may have set my self up with that title, implying that I will have more thoughts later on in this process. Likely but not guaranteed.

It is now a little more than 3 weeks since I was let go from Food52. While I very much appreciate the advice I got from friends to take a break and rest before I jumped into a job search, that is not who I am. My anxieties would not let me sit in a state of unemployment without trying to make some progress towards finding a new job. It has become clear that I, like I am sure many, define a large part of their being based on their career. While I have not enjoyed the emotional rollercoaster of self worth, it is a good lesson to learn. I look forward to the day when I can retire and I need to make myself ready to define myself through the activities I love that are not my career.

While I wanted to start making progress I did not want to open the floodgates right away, so I have a roadmap of sorts for the finding roles that I could be interested in and apply for them:

  1. Share my situation with my network and see if anyone has any openings
  2. Find companies that I respect and look for openings there
  3. Reach out to recruiters that have contacted me in the past with interesting roles
  4. Mark myself as “available for work” on LinkedIn
  5. ???

At this point I am still in phases 1 & 2 which pretty much happened simultaneously. My network included friends, former coworkers, and the job boards on a few Slacks that I am a member of. I was not surprised to get compassion and positive feedback from my friends and former coworkers, but what did surprise me was the response from the various Slack communities I am a part of. While I participate in these groups, I am not very active an some of them are huge so I had not been expecting the reception my message got.

I was overwhelmed by the positive and supportive response from these communities. I had numerous people reach out with ideas and several with specific opportunities that they encouraged me to apply for. People were willing to schedule calls to talk about roles and give their time to help me, basically a stranger. This was truly touching and I can’t give my thanks enough.

A choice I have to make in my process is about the size of the company I want to work in. A criticism I had of my time at Food52 was that as the senior-most engineer I did not have a community to work with nor did I have a clear path of growth. I am finding in the roles that I apply to that I have a choice of looking at larger organizations that could bring with them a community of Staff+ engineers to work with. And that idea is very tempting. The other side is to look at the smaller companies and know that I could have a large influence on them in a broad set of topics. I am not ruling either out but it is interesting to me that the difference has been rather stark in the companies I talk with.

So at this point I am in later stages with a couple of opportunities that I am pretty excited about. My hope is that I do not have to advance to stages 3 and beyond, but I should know more in the coming weeks.


The only constant in life is change, they say. The last few months have definitely been that way. But change is an opportunity for new beginnings and for me, my time at Food52 has ended and I am starting the journey to find a new work home. Over the past few days I have been pondering what I want to do and with whom I would like to work. I’d like to share some of my thoughts here as I work through them. This change is exciting, scary, humbling, and even a bit confusing as I come out of nearly 8 years at Food52 (ok, 7 years, 10 months, 28 days if I am being pedantic).

The coincidence with the new year is not lost on me, a time of reflection on what direction I want to proceed in. Through my career I have worked for small companies and large, public and private, and as an individual contributor and in engineering management. Most recently I was a Principal Software Engineer. I appreciate this kind of role as it kept me hands-on in coding but also has some of the benefits of engineering management: mentorship, guiding technical direction, and visibility outside of the engineering organization. However, I have run engineering teams before and there are aspects of those that can also be very fulfilling within the right kind of organization.

At this moment, my focus is going to be mostly on Staff/Principal IC roles with the occasional management role thrown in if the company seems like a good fit. I’ve started writing down my wishlists which will certainly evolve over time. I am not really focused on the particular technology facets of the companies I am looking at, it is really the people and culture that matter to create a fulfilling environment.

Must haves:

  • remote friendly, preferable remote native
  • a mission that I believe in and is actually practiced day to day
  • solid DEI practices
  • teams that respect all of their engineers regardless of seniority
  • a respect for work/life balance
  • open and communicative leadership

Nice to haves:

  • missions that align with my personal interests: outdoors, travel, cooking
  • the ability to cross train in adjacent technology that I am not great at (yet)
  • small to mid size organizations (or well defined units within a large company)

No thanks:

  • business models based on manipulation of users’ behaviors
  • “hardcore”

If you have made it this far and would like to learn more about me, here are some useful links:

Here are some great resources I have found for aiding in a job search.

  • Angela Riggs’ Job Search Template offers a simple but highly effective template in Trello for keeping track of roles you are considering and where you stand in the application and interview process for each. It took what was already a mess of emails and notes I had been building at this early stage and made it much more manageable.

  • Interviewing is like speed dating, you only have small blocks of time to decide if the company you are talking to is a good fit for you. It reminds me of one of my favorite lines from Neal Stephenson’s Snow Crash: “Condensing fact from the vapor of nuance.” Charity Majors (who you should be reading in general) has a great article of questions to try to find those signals as you interview: How can you tell if the company you’re interviewing with is rotten on the inside?.