The bumpy road to patching

I basically took most of Wednesday off this week. My wife had a bit of fever and that's a good excuse for me to spend a day playing with our two children (and their two cousins). I was planning to get some work done in the afternoon, but we went to the beach, had great fun in the waves on a windy day and stayed longer than expected. Later we made cinnamon buns – still no time for work..

I'm not in Oslo right now – so actually, I'm living in a house without an internet connection. I walk about five minutes to my mother-in-law's house to get online. So on this particular day, I was offline until we went to my mother-in-law for a family dinner and I wanted to check E-mail quickly before walking the children home to bed.

And – OMG. There were bugmail, E-mail and Twitter messages saying Hotmail is broken.
Apparently Microsoft was busy pushing out updates while I wasn't paying attention.. :eyes: :yikes: I basically don't care which of Yahoo mail, GMail and Hotmail is the world's #1, #2 or #3 webmail – if any of them breaks in Opera it's a major crisis.. Seems I had some work to do :irked: .

So I walked the children back home, put them to bed, stayed until they were asleep and went back to the other house and my laptop to have a look at the problems.

The parse error caused by this use of "in" inside a "for" condition

for(i.isArray(t)||(t in e?t=[t]:(t=i.camelCase(t),t=t in e?[t]:t.split(" "))),s=0,c=t.length;s<c;s++)

was already analysed, but lots of buttons in the UI were unclickable for unknown reasons. Apparently click events were not even fired. Perhaps a problem with either registering event listeners or dispatch and propagation of events?

Now, unwinding this code to understand it might take a while even before noon:

i.event={add:function(n,r,u,f,e){var v,h,a,w,p,o,b,l,y,k,c,s;if(!(n.nodeType===3||n.nodeType===8||!r||!u||!(v=i._data(n)))){for(u.handler&&(y=u,u=y.handler,e=y.selector),u.guid||(u.guid=i.guid++),,a||({}),h=v.handle,h||(v.handle=h=function(n){return typeof i!="undefined"&&(!n||i.event.triggered!==n.type)?i.event.dispatch.apply(h.elem,arguments):t},h.elem=n),r=i.trim(wt(r)).split(" "),w=0;w<r.length;w++)p=kr.exec(r[w])||[],o=p[1],b=(p[2]||"").split(".").sort(),s=i.event.special[o]||{},o=(e?s.delegateType:s.bindType)||o,s=i.event.special[o]||{},l=i.extend({type:o,origType:p[1],data:f,handler:u,guid:u.guid,selector:e,quick:e&&ef(e),namespace:b.join(".")},y),c=a[o],c||(c=a[o]=[],c.delegateCount=0,s.setup&&,f,b,h)!==!1||(n.addEventListener?n.addEventListener(o,h,!1):n.attachEvent&&n.attachEvent("on"+o,h))),s.add&&(,l),l.handler.guid||(l.handler.guid=u.guid)),e?c.splice(c.delegateCount++,0,l):c.push(l),[o]=!0;n=null}}
dispatch:function(r){r=i.event.fix(r||n.event);var v=(i._data(this,"events")||{})[r.type]||[],y=v.delegateCount,k=[],0),d=!r.exclusive&&!r.namespace,p=i.event.special[r.type]||{},b=[],e,w,f,c,a,l,s,h,u,o,g;if(k[0]=r,r.delegateTarget=this,!p.preDispatch||,r)!==!1){if(y&&(!r.button||r.type!=="click"))for(c=i(this),c.context=this.ownerDocument||this,;f!=this;f=f.parentNode||this)if(f.disabled!==!0){for(l={},h=[],c[0]=f,e=0;e<y;e++)u=v[e],o=u.selector,l[o]===t&&(l[o]=u.quick?rf(f,u.quick),l[o]&&h.push(u);h.length&&b.push({elem:f,matches:h})}for(v.length>y&&b.push({elem:this,matches:v.slice(y)}),e=0;e<b.length&&!r.isPropagationStopped();e++)for(s=b[e],r.currentTarget=s.elem,w=0;w<s.matches.length&&!r.isImmediatePropagationStopped();w++)u=s.matches[w],(d||!r.namespace&&!u.namespace||r.namespace_re&&r.namespace_re.test(u.namespace))&&(,r.handleObj=u,a=((i.event.special[u.origType]||{}).handle||u.handler).apply(s.elem,k),a!==t&&(r.result=a,a===!1&&(r.preventDefault(),r.stopPropagation())));return p.postDispatch&&,r),r.result}}

Being sleepy doesn't exactly speed up one's analysis of foreign code. :zzz: I wish I could tell you what sort of dreams one has when taking naps deep inside Microsoft's event handling, but I don't really recall them. Probably I dreamt of snakes, sneaking through the stacks to call jQuery.delegate(). (I'm seeing more snakes this summer than entire last year – I guess having to walk five minutes to get online in the morning sort of causes it – and as I'm not really used to snakes they tend to show up in my dreams too). Long story short, in the early hours of the morning I still had not fully understood the problem but I realised returning 0 for event.button in click events solved it. So around 6 o'clock A.M. I quickly went through some pending patches, activated them and started signing new browser.js files, sent those off to the update server admins and saved a draft announcement blog post before I went offline. I was supposed to teach one dance class and a workshop that Thursday, and had to prepare. Luckily it's not possible to be sleepy while dancing.

So, in a different time zone my server admin colleague wakes up and pushes out the new browser.js. My only excuse was that it was compiled and tested in a pretty sleepy state. It had a bug. A pretty bad one..

I've been tweaking some browser.js basics these days, among those site detection. Browser.js now defines a hostname.endsWith() method, and on Thursday it looked like this:

endsWith:function(str){ return this.value.length==this.value.indexOf(str)+str.length; }

It also included a brand new site patch for which looked like this:

if(hostname.endsWith('')){			// PATCH-769, Opera throws when XSL variable has disable-output-escaping attribute, breaks sorting on
		var xhrDocGetter=(new XMLHttpRequest).__lookupGetter_responseXML;
		XMLHttpRequest.prototype.__defineGetter__('responseXML', function(){
			for(var elms=doc.getElementsByTagName('*'),elm,i=0;elm=elms[i];i++){
			return doc;

The endsWith method uses indexOf() but forgets to take into account that the value indexOf() returns for "not found" is -1. The '' string is 11 characters long. Hence,


turns into -1+11 – which does of course result in the number 10. If the site's server name was exactly 10 characters long, endsWith() would happily conclude that because -1+11 equals 10, the site you're on is one that ends with And what would applying the staples patch do to a random site?

Well, if the site used Ajax and XMLHttpRequest, and requested data that was plain text or JSON or some other non-XML format, but ran a script that tried to read responseXML, the doc.getElementsByTagName() call would throw because doc would be null. Most libraries will try to read responseXML to see if there is XML data coming from the server, and none of them will expect it to throw an exception.

The net result of that? After the browser.js update, ajax stopped working in Opera on every site with a server name that is exactly ten characters long. :doh:

Co-browser.js-admin Ola was on vacation, and I wasn't online again until Friday morning. :insane:

Now, that Friday I was supposed to travel. I checked E-mail quickly on somebody else's computer, saw the problem and sent update server admin an E-mail asking them to roll back the update until I could sign new files.

Then I spent 1 hour on a bus and 4 hours on a train, trying to sleep. Some people claim that counting sheep is a good way to fall asleep. I can testify that counting characters in domain names isn't. My brain kept itself busy trying to figure out how many of the web's major destinations have a ten-character name.. = 14, = 12, = 11 (phew!), is exactly 10 but I don't think they run any services there. I can live with breaking demos on – that doesn't really count as breaking the web, it's merely wildly embarassing. But ..ouch, what about ..? :yikes:

When I arrived it didn't take long to fix those bugs and sign new files. To sum up the post mortem – we'll probably add stricter code review measures for browser.js, and have better emergency rollback measures when we mess something up anyway.

Avoiding debugging while sleepy and making sure every member of the family stays healthy around the clock would also help, but I'm not sure how to implement that without regressions.


14 thoughts on “The bumpy road to patching

  1. Hope your wife's feeling better, and thanks for avoiding meltdown with Outlook not working on Opera 12.Your efforts are really appreciated.Now if we can just have a fix so we can upload files to Skydrive (just sits there and doesn't upload anything) & to be able to make attachments to emails (which don't attach anything at all)we could use Opera 12 with Outlook rather than using (the actually rather good) Firefox Aurora. One web and all that.Sorry to be a pain.Mind you though, if bees had knees then they'd have Opera 12 on them. It's really really good! Thanks:cool:

  2. Originally posted by Mickeyjoe-Irl:

    Drowsy coders only kill websites

    Thanks for putting things into perspective :love:

  3. Wait, you left your wife at home with a fever to goto the beach?And how in the world do you not have a smart phone with email?

  4. Originally posted by toyotabedzrock:

    Wait, you left your wife at home with a fever to goto the beach?

    Yes, nothing heavy, the children already had it and recovered and she was getting better already ;)Originally posted by toyotabedzrock:

    And how in the world do you not have a smart phone with email?

    Various reasons, I've been so stingy and backwards-looking I've mostly tried to get by with feature phones, I don't want to pay roaming data charges for email abroad, and I don't usually need email on my phone when I spend a lot of time online anyway. 🙂

  5. Originally posted by hallvors:

    Various reasons, I've been so stingy and backwards-looking I've mostly tried to get by with feature phones, I don't want to pay roaming data charges for email abroad, and I don't usually need email on my phone when I spend a lot of time online anyway.

    I vote for Opera to provide you with a mobile broadband card for your laptop and cover the cost of the data usage. Great investment for Opera to make. 🙂

  6. Despite his critical role at Opera HQ, I think it's excellent that Hallvord gets some time away from Opera, and that he doesn't have emails chasing him all the time.—Thanks for what you do for all Opera users, Hallvord. I'm glad your wife and children have a higher priority to you than us. They will always be the more important investment of your time. I know you already know that. I just like to encourage dads being good dads. It's good to read that there'll be enhancements to the browser.js refinement process that will allow your adrenal glands to function in the normal range. I agree with Rafael Luik in terms of your blog entry being art. I like how you don't just say the basic facts, but that you add your personality into your accounts of your trouble shooting sessions.What style of dance do you teach?

  7. I think you and Ola should not be on vacation at the same time ;)As always, nice read and valuable information from inside sitepatching 🙂

  8. Originally posted by ouzoWTF:

    you and Ola should not be on vacation at the same time

    We weren't 🙂 I've been working throughout Ola's vacation, though not in Oslo ;)Originally posted by BridgeBuilderKiwi:

    What style of dance do you teach?

    I've taught both classical ballet, contemporary dance and a more creative/contemporary/dancer-driven workshop over the last few weeks. Teaching is great fun and a pretty rare learning experience at the same time.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s