Launching OddPatch 0.1 Beta


Launching in browser.js tomorrow or so: OddPatch beta, solving many of the compatibility issues with the service formerly known as OddPost, now Yahoo!Mail beta.

It's a complex patch sorting out a daunting number of issues on both sides. When things get this complex, it's rarely only "their fault" or "our fault" – the testing has uncovered several bugs in Opera and several mistakes in their JavaScript. Here's a walk-through of the entire patch:

// browser sniffing workaround - walking in through the back door
if( location.href.indexOf( '/dc/system_requirements?browser=blocked' ) >-1){
location.href='/dc/launch?sysreq=ignore';
}

Y!Mail has three modes for browsers: supported, possibly working, or blocked. Opera is blocked, but luckily there is a backdoor that will bypass the sniffing.

The irony here is that when they block us, they are making their work on Opera-compatibility much harder than necessary. If we get access, we'll do our best to make things work: test, find bugs, even decide to support things we haven't supported previously (we'll have selectSingleNode soon because Y!Mail uses it heavily, and their code was also a very important reason why DOM2 Style support was prioritised for Opera 9.0!). Blocking us makes it much harder for us to make their life simpler.

if( top.location.href.indexOf('/dc/launch')>-1 ){ 
// Gecko compatibility library uses defineGetter and defineSetter. We need to fake them.
//* Patch below is required but causes trouble.. 
Object.prototype.__defineGetter__=      function(){}
Object.prototype.__defineSetter__=      function(){}

This is known as "fake it until you make it". We won't have getters and setters anytime soon, but things will work anyway if we pretend we do.

// IEism called loadXML, basically a DOMParser / DOMLS equivalent
// must handle XML fragments without root element!
Element.prototype.loadXML=function(s){  
try{
var d=new DOMParser().parseFromString(s, 'text/xml'); 

This, I think, is a bit of the strange world IE lets you into if you put an XML tag in a page. That tag takes on a life of its own and starts behaving in some contexts like a document, it aquires several methods and properties – and though I at first thought I could simply fake it with a DOMParser I had to think again because…

}catch(e){ // DOMParser could not parse fragment, probably because of missing single root element. Workaround time..
var d=document.implementation.createDocument('', this.tagName, null), el=d.createElement('el');
//?? why did I use this.tagName there?
el.innerHTML=s;
for(var i=0 ; i<el.childNodes.length;i++){
d.appendChild(el.childNodes[i].cloneNode(true));
}
}

Yes, they are not always playing with well-formed XML fragments. Oh well, we'll pull out good-old-tagsoup-parsing .innerHTML and eat their strings anyway. Then we move on and fill in some other required bits and pieces of IE's XML DOM. Right now I'm not sure if all the stuff in this block is required, but there it is.. I'll be the first to admit that both code and comments are evidence of the somewhat chaotic process of late-night patching..

// faking IE-style XML element DOM - separate documents with documentElement within the main doc's DOM
this.documentElement=d.documentElement||d.firstChild; 
//?? firstChild is probably leftover from earlier versions using documentFragment?
this.XMLDocument=d;
// address book loading checks .parseError.errorCode
this.XMLDocument.parseError={ 'errorCode':0 };
return d;
}

And then is a peculiar mystery, who would ever need a function called isSameNode when == would presumably do the job?

// some method called isSameNode is called. Not sure where it comes from but simple enough to fake..
Element.prototype.isSameNode = function(n){
return n===this;
}

Here we go, more delicacies from IE's internals: the handy XPath method selectSingleNode. I have quite some reservations against loadXML and the other XML DOM stuff above, but selectSingleNode should be written into a standard as soon as possible because document.evaluate needs too many arguments for lazy JS coders and the returned object is too fiddly too.

// selectSingleNode support
var realSelectSingleNode=function( expr, resolver ){
var result=(this.ownerDocument?this.ownerDocument:this).evaluate( expr+'[1]', this, resolver, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,null );
return ( result && result.snapshotLength ) ? result.snapshotItem(0) : null;
}
Node.prototype.selectSingleNode = function (expr, resolver)
{ 
  if (!resolver)
    if (this.nodeType == Node.DOCUMENT_NODE){
      resolver = document.createNSResolver (this.documentElement);
    }else if(this.nodeType == Node.ELEMENT_NODE && this.ownerDocument && this.ownerDocument.documentElement ){
      resolver = document.createNSResolver (this.ownerDocument.documentElement);
    }else{
resolver = document.createNSResolver (this);
    }
  return realSelectSingleNode.apply (this, [expr, resolver]);
}

Now, up till now the patching has been quite ordered. A patch is a bit of a kludge anyway, but so far it's been a nice kludge. Here come the problems that were too ugly for a nice kludge. Certain issues just required search and replace operations on the script source code, nothing else to do about them – and because Yahoo source is compressed and variable names random, search and replace must take that into account and go for seriously complicated and ugly regular expressions.

opera.addEventListener('BeforeScript', function(e){
// This is the riskiest patch
// Fixing typo: missing ' after attribute value
e.element.text=e.element.text.replace( /):(">")),/, "):("'>"))," );
e.element.text=e.element.text.replace( / id='_test_add_folder>/, " id='_test_add_folder'>" );

Yep, twice in their source code they say things like <tag class="foo> omitting the closing quote. That caused Opera to parse it as a text node instead of an element, meaning source code would appear here and there in the interface. Oops.

// WebForms2 problem: button attribute "action" is a URL in WF2
e.element.text=e.element.text.replace( /.(actionb)/g, ".js$1" );

Specs and implementations collide again: if you set input.action to a value like 'markAsSpam' the WF2 spec means it will be resolved as a URL, so when the script reads it again it will see 'http://mail.yahoo.com/dc/markAsSpam‘ which is not at all what it expected.

// send button not working - attribute nodes must be in the document they will be used
e.element.text=e.element.text.replace( /(w).setAttributeNode((w))/, "$1.setAttributeNode($1.ownerDocument.importNode($2, true))" );

Hm, is Firefox sloppy with exceptions on cross-document node usage again?

// workaround for getting the documentElement.xml  markup
e.element.text=e.element.text.replace( /(([w.]*)documentElement).xml/g, "(document.implementation.createLSSerializer()).writeToString($1)" );

IE's XML DOM rides again. Elements over there have an .xml property which is basically the equivalent of .innerHTML for an HTML element, showing the inner serialised markup.

I didn't take a long and hard look at how Y!Mail used it but some of that seemed very weird. I had the impression that they read .xml of the contacts list only to pass it around as a string and use loadXML later on. Why would they serialize markup just to parse it right into a DOM tree again? Oh well, there is probably some complex reason..

// for...in on objects run into our faked __defineGetter__ and __defineSetter__
// we try to add an exception to any for...in loops
e.element.text=e.element.text.replace( /(for((var |)(w*) in w*){)/g, "$1if($3.match(/^__define/))continue;" );

It turns out "fake it until you make it" wasn't such a good idea after all. The site called our bluff with code like

function foo(obj){
for( p in obj )return false; return true
}

var bar = {}; if(!foo(bar)) return;

and what exactly they meant by that I don't know either, except to check that a newly created object really REALLY had no properties. Huh?

// To: / CC: autocomplete fails
// we support IE's TEXTAREA.createTextRange but unfortunately not its boundingLeft property. Improving object detection..
if(e.element.src&&e.element.src.match(/ac.js$/))e.element.text=e.element.text.replace( /if ( editCtrl.createTextRange )/, "if ( editCtrl.createTextRange  && editCtrl.createTextRange().boundingLeft )" );

This is a typical trap of piecemeal implementation of something: we support whatever of IE's stuff was deemed important to get some plaintext formatting JavaScript to work with 8.x. .boundingLeft wasn't on the list back then. Sorry. Look a bit harder when you look for something.

// Preferences not read correctly from XML attributes
// IE has an attribute node .text property. .nodeValue will work in Opera..
e.element.text=e.element.text.replace( /.selectNodes((w*));s*}return((w*).length)?(w*)[0].text:/g, ".selectNodes($1);}return($2.length)?$2[0].nodeValue:" );

IE again. text alias nodeValue, enough said.

// We throw an unwanted exception if both arguments to insertBefore are the same node
e.element.text=e.element.text.replace( /var (w*)=(w*)?(w*).nextSibling:(w*).firstChild;s*(w*).insertBefore((w*),(w*));/, "var $1=$2?$2.nextSibling:$4.firstChild;if($6!=$1) $4.insertBefore($6,$1);" );

Now this is plainly a bug. The DOM spec says we should throw an error if the node you insert is a parent of the reference child, but we also did so if the inserted node was the reference child itself.

The code doesn't make sense, mind you… Why do you want to replace an element with itself?

// Opera 9.00 and 9.01 has a bug that means createContextualFragment on table elements is unreliable
// easily the worst patch.. but then it works around a really tricky bug..
if( navigator.userAgent.indexOf('9.01')>-1 || navigator.userAgent.indexOf('9.00')>-1 ){ 
// UA detection to target specific bug in specific version is OK
e.element.text=e.element.text.replace( /(b(w*).selectNodeContents((w*));s*var (w*)=(w*).createContextualFragment((w*)))/, "if($3.tagName=='TBODY'||$3.tagName=='TR'){ $2.createContextualFragment=function(s){var n=s.match(/<(\w*)/)[1]; var e=document.createElement('div');e.innerHTML='<table><tbody>'+s+'</tbody></table>';return e.getElementsByTagName(n)[0];  } }$1" ); 

Just as ugly as it looks, just an attempt to make the code work in 9.01.
Imagine if Yahoo!Mail blocked us until they one day decided that it was necessarily to start working on Opera compatibility? If we never had gotten to test their system, they would probably have to add such an ugly workaround to their application to get around this bug. Developers everywhere: please, don't sniff, just leave more of the burden of compatibility on the UA's table (and listen to feedback!).

Hey, we're done with the replacements! It wasn't pretty, and I look forward to deleting one by one while things are fixed on either side. That will also give us a nice performance lift. Their scripts are huge. At some point I did some profiling of the above replace calls and found that up to half of the time it took Opera to load Y!Mail was spent applying the above patches.

}
}, false)

// No scrollbars appear for message list..
// uses an "overflow" CSS property to control scrollbars. e.overflow="-moz-scrollbars-vertical", and some odd clipping as well..
document.addEventListener( 'load', function(){ setTimeout( function(){ var divs=document.getElementsByTagName('div');for(var i=0,div;div=divs[i];i++)if(div.className&&div.className.indexOf('fakeScrollBar')>-1){div.style.overflow='auto';div.style.clip='auto';}},500);}, false );

Yes. That problem. It uses some CSS I still haven't fully understood.. I think they wanted to show an element with a scroll bar but clip the whole element away so only the scroll bar would be visible.

// redraw problem hides To: field in compose screen
document.addEventListener( 'load', 
function(){ if(top.document.frames['newmessage']){
setTimeout( function(){try{top.document.frames['newmessage'].document.body.className+=' ';}catch(e){}},1000);
}}, true);

This is another tricky one, it has to do with timing and I still haven't quite captured the sequence of events. Basically the "To" and "Subject" fields in the compose screen disappear until you click "Show BCC".

// sluggish performance due to unintended event capture
(function(ael){ 
window.addEventListener = function(type, func, capture){ ael.call(window, type, func, false); }
})(window.addEventListener);

..and just to top if off, they had to capture events by mistake. Of course. That's from the curriculum of "How to code Opera-incompatible websites 101".

opera.postError( 'Yahoo mail patched' );
}
}

Yippee! We did it!

Now, this patch is (repeat after me) in beta! There are problems that I'm aware of but haven't fixed, and there are problems that I'm not aware of and haven't fixed. And while I was working on this stuff, code changes in Yahoo mail would break things again every few days (and even break differently on the U.S. and the U.K. sites!). So, hurry up and try it while you have a chance! I'll try to keep the patch maintained, and we sure hope that we'll get all the issues sorted out from either side for a fast, friendly, responsive experience – somewhere in the future..

Advertisements

15 thoughts on “Launching OddPatch 0.1 Beta

  1. ..and if some overnight update means the patch doesn't WORK when browser.js comes out tomorrow I'll just go demonstrating against something. whatever..

  2. Reading between the lines, it sounds to me like even though you guys have a relationship with Yahoo and do have contacts with them, that they pretty much excluded you from the conversation about this site. Otherwise, I don't think you would have attempted all the work it took to come up with a Frankenpatch.But why would they close off? Their developer site gives Opera 9.0 under XP an overall "A" grade for compatibility with Yahoo sites (though not this one), so shouldn't they have some interest now? Maybe in the old days I would have expected that attitude, but not today.It's also interesting that Yahoo itself was willing to do all the heavy lifting (probably) necessary to get Firefox/Mozilla/Netscape going.

  3. Hoo boy! Reminds me of when I wrote Proxomitron patches to get Gmail to work with Opera 7/8! :yuck:Thankfully, from the looks of things, Gmail was a whole lot simpler! Very impressive Mr Steen 🙂

  4. Congratulations Hallvord!Impressive work.Yet, it's really bad Yahoo doesn't fix their site. Most performance penalties are in the regex.

  5. xErath: thanks for your thorough review of the patch, several improvements will appear in the next browser.js release 🙂 Regarding maintaining browser.js for Opera 8.x: that is sadly not likely to happen with any regularity. browser.js is by its very nature a high-maintenance nightmare and being the wonderful magic it is we are sorely tempted to use it across platforms and products beyond desktop – just imagine how many versions of it we are going to have to test.. 😦 I can of course pretend I'll try to maintain the 8.x version but in reality I won't have time.rseiler: we have a good relationship with Yahoo, and they do fix things for us – but we also have to live with being at the bottom of the priorities list for something like Yahoo mail beta. It's descending from a system that was written to be IE-only after all, now they have been tweaking it for Gecko compatibility and I guess Safari is next on their list. It is the vicious cycle of low usage share causing incompatible services which again prevents users from switching to Opera. browser.js can break that cycle, and when something is as big as Y!Mail it can make a real difference. 🙂

  6. I couldn't make may way through all that JS :)One question, Hallvors – does this JS patch (it looks identical to the Yahoo Mail Beta patch in browser.js) supposed to allow pane resizing? Or is that something that still needs work?

  7. does this JS patch (it looks identical to the Yahoo Mail Beta patch in browser.js) supposed to allow pane resizing?

    Not yet. It's a known issue, and there's a patch for it.Basically, they try to manipulate a stylesheet in another domain, which fires the expected security exception.

  8. Yahoo Mail Beta receives update.Some interesting answers from Yahoo guy under rckenned nick name:Originally posted by rseiler:

    …Anyway, when are you guys going to get around to supporting Opera? Opera has implemented a hack to make it kinda-sorta work, but it's dog slow and prone to problems.

    Originally posted by rckenned:

    I'm not sure what the official line is on Opera. I work on the web service team, so I'm not sure what the frontend guys have up their sleeves in terms of browser support. Even if I did, I probably wouldn't be allowed to discuss those plans just yet.I do know that some people have gotten Opera 9.1 to load Yahoo! Mail Beta and say it runs great. They're all on the new release that we just put out, so maybe when the update reaches your account you'll see the same thing.

    Originally posted by FataL:

    It would be nice if you guys finally take care about Opera too.

    Originally posted by rckenned:

    The fact that I've seen people get it running in Opera 9.1 makes me think it won't be long for Opera support. It really seems like the Opera guys are stepping up and filling in a lot of missing functionality in their browser. That makes it much easier on the application developers.

    So, let's hope some things was fixed!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s