25 April 2025

Another war story: the hardest bug I ever debugged

A couple of weeks ago I came across this fantastic post from Jacob Voytko describing a nightmare bug he analysed while working on the Google Docs team. I love a good bug hunt, and Jacob’s story reminded me of one of my own. Since I enjoyed his post so much, I figured I’d share mine.

At the time I was working at eyeson where we were building a video conferencing app. I worked on the browser client of the product and one day we received a bug report that went something like this:

Hi,
I like your service, but sometimes, when I join a call, my camera is rotated by 90 degrees.
Cheers

(It has been eight! years, so the details are a little fuzzy.)

Essentially, the person would be joining a meeting like this:

rotated webcam Image created with ChatGPT.

After replying to ask about the user’s setup, my curiosity got the better of me and I started digging.

The obvious place to start - the camera code

First place I had a look at was the code to acquire the camera stream.

If you’ve never worked on a WebRTC application, acquiring a user’s camera, is actually quite simple. It boils down to something like:

getUserMedia({ audio: true, video: true, });

Where one can supply different “constraints” to getUserMedia. Our constraints were a bit more specific - we remembered the users selected camera and set some preferred resolutions,

getUserMedia({
  video: { 
    deviceId: { exact: chosen.deviceId },
    width: { ideal: 720 },
    height: { ideal: 1280 },
  }
})

but we didn’t supply any to change the orientation (I’m not sure that is even supported). I didn’t spot anything obvious in our code and moved on to trying to replicate the issue instead.

Trying to reproduce it

I had a setup like this:

rotated webcam Image created with ChatGPT.

My setup included a MacBook with an internal webcam and an external camera mounted on a second monitor, plus a Dell PC with a similar dual-camera setup. This way I could hop into meetings with different computers, browsers (Firefox, Chrome, Safari, Edge), cameras and OS’s rather quickly.

As you can already tell, there are quite a lot of variables, and I spent a good chunk of time testing different camera/browser combinations — no luck.

Fortunately the user reported back and said they were on windows, using Edge and only had one (builtin) camera. I was pretty sure I had already covered that combination but I tried again anyways and couldn’t replicate it.

Digging deeper into meeting flows

We had several ways to join a meeting: invitation links, fixed room URLs, dashboard shortcuts, a preview screen where you could select your devices and make your hair before actually joining and more.

The fastest way to get into a meeting was by sharing a link to a room and since that was also the most convenient one for developers, I had only tested that flow.

invitation page Image created with ChatGPT.

Now I went through the others and lo and behold, at some point, when clicking on an invitation link to the meeting and selecting my internal camera, I could see the picture being rotated by 90 degrees.

I finally reproduced the issue in our app, but still had no idea what caused it. At that point, I was pretty sure it was an Edge bug.

Building a minimal repro

So I started replicating our flow step by step with the intention of providing a sample repo (the repository is on GitHub btw but spoilers).

I created two HTML pages, one for acquiring the camera stream (usermedia.html) and one for joining the meeting (join.html). usermedia.html had a simple getUserMedia call like the one I shared above and join.html had a link to join.

<a href="/usermedia.html">Join</a>

But after clicking the link, the camera looked fine. I compared the invitation link with the one we used and noticed that I was missing the target.

<a href="/usermedia.html" target="_blank">Join</a>

Yet again, no luck. I did notice though, my external camera was being picked by default so I added some code to ensure that the internal one was selected.

navigator.mediaDevices.enumerateDevices()
      .then(gotDevices)
      .then(getUserMedia)
      .then(gotStream)
      .catch(console.error);

    function gotDevices(mediaDevices) {
      console.log('gotDevices: ', mediaDevices);
      const cameras = mediaDevices.filter(d => d.kind === 'videoinput');
      const camera  = cameras.find(c => c.label.match(/integrated/i)) || cameras[0];
      console.log('camera.deviceId: ', camera.deviceId);
      return camera.deviceId;
    }

    function getUserMedia(cameraId) {
      const constraints = {
        video: { deviceId: { exact: cameraId } },
        audio: true
      };
      console.log('getUserMedia: ', constraints);
      return navigator.mediaDevices.getUserMedia(constraints);
    }

    function gotStream(stream) {
      console.log('gotStream: ', stream);
      let videoElement = document.getElementById('video');
      videoElement.srcObject = stream;
    }

Once more, everything looked fine. I was able to replicate the issue in our app consistently, but not in my sample app. Something was still missing.

Quick aside

At that point I had spent an unreasonable amount of time on this. We only had a single report of that issue. But I love a good bug hunt. These kinds of debugging sessions, while frustrating, and time consuming, are sooo much fun and very satisfying when you finally figure it out. I simply cannot, for the life of me, “let it go” and move on to the next thing.

The missing piece

Anyway, onto the next step. I started reviewing the app flow again and noticed something I’d missed earlier. When clicking the link to the meeting we actually got redirected (301).

So I added that part as well

import BaseHTTPServer, SimpleHTTPServer

IP='192.168.0.25'
PORT=4444
LOCATION=("http://%(ip)s:%(port)d/usermedia.html" % {'ip': IP, 'port': PORT})

class myHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
  def do_GET(self):
    if self.path == '/redirect.html':
      self.send_response(301)
      self.send_header('Location', LOCATION)
      self.end_headers()
    else:
      return SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)

httpd = BaseHTTPServer.HTTPServer((IP, PORT), myHandler)
httpd.serve_forever()

And finally, the camera was rotated by 90 degrees. I could replicate the issue consistently in my sample app. Finally.

Recap

To recap, the bug only occurs:

on Windows with the
Edge browser
after a click on an href with target=“_blank”
which results in a redirection (Moved Permanently 301) to
a page requesting the users internal camera (getUserMedia)

It is

Not reproducible with an external camera
Not reproducible when opening the link in the same tab
Not reproducible when opening the link in a different tab without redirection

But what the Edge is going on?

After spending so much time and effort on this, the final step was to file a bug and hope to learn from the Edge team what was going on. So I did. I provided the repo, wrote up a summary, filed the ticket - and then I waited. And waited.

At the time, Edge was still based on EdgeHTML and only available on Windows (I think). Unfortunately the ticket never got any traction or comment, and at some point, Edge migrated to Chromium. To this day, I’m still curious about what was going on. But I guess, we will never know.

Moral of the story?
¯\_(ツ)_/¯ - Enjoy the hunt but don’t expect closure.