the backtrace of an Y!Mail debug session

Advance warning: this blog post dives into the most complex bits of Yahoo Mail beta. Along the road I'll explain a few tools, tips and tricks I rely on when analysing problems. I'm afraid you may not understand much of this post without prior knowledge of JavaScript. Feel free to read it anyway, but you've been warned!

The other day I claimed to know why Y!Mail was unusable in Kestrel, but even as I posted that I had a feeling I didn't have the full story yet. Yes, I had found some incompatibility between Opera and Firefox that DID trip things up and made confusing "Generic error" messages appear. I even found the Mozilla bug report saying Moz's behaviour was a bug (and by extension that Opera's was correct) – but still, this seemed more like a symptom than an actual cause. Because even if Firefox didn't throw exceptions, how could Y!Mail get away with sending us so much invalid XML in the first place? Some of that XML contained information that WAS available inside Y!Mail when running in Firefox. Was Y!Mail's XML invalid only in Opera? If so, why?

So next day I knew understanding this issue was going to be top priority, no matter how much I'd like to procrastinate with simpler bugs before diving into Y!Mail source code.

For a background, I'll recap the initial debugging from the previous day. The problem was the "Generic error" message on the screenshot above on the right, which appeared on Y!Mail load and caused the whole Y!Mail to be unusable.

First step with code like this is always making the all-on-one-line source more readable with one of the wrap scripts available. Luckily it is very simple to make Opera use the wrapped source instead of the original: open the script's URL in a new tab, view source, copy it and do the wrapping, paste it back to the editor, save and use Opera's "Reload from cache" setting.

The reload from cache menu entry may not look very interesting, but it is currently your best friend when debugging problems in Opera. It allows you to insert any sort of debug code from your source editor and study the output almost instantly. That is very useful. (It does have bugs though, IFRAME contents is usually re-validated from server, and scripts may also be – I guess some part of Opera respects expiry and cache-control headers even for "reload from cache". It usually works but I have to keep an eye on the HTTP requests just in case.)

However, before you can insert any debug code, you have to find the part of the JavaScript that contains the problem. With something as complex as Yahoo Mail, that's not easy. The error message doesn't make it easier: just "Generic error" without a backtrace – hey Opera, we're old friends but I know your shortcomings in the debugging department better than any griping, Firebug-spoiled web developer. That error is so unbelievably unhelpful, did you know?

For searching through code and many other debug tasks, I highly recommend the HTTP debugger Microsoft Fiddler. If you do any sort of web application or web browser development, this tool is a must-have. Below is a screenshot of Fiddler showing parts of the traffic that occurs during a Y!Mail session, the selected session is from an XMLHttpRequest and you'll see how Fiddler shows you the markup that was sent to the server (top right pane) and the markup that was returned (bottom right):

As you see, Fiddler keeps a record of all the code a website loads or sends – and you can search through all that code at once! To attack a problem like this I might try to search for a part of the error message:

The response seems promising:

The files that contained this message are now highlighted, so I can pick one of them and use the inline find field in the source pane to find the actual location:

Here's a match:

	function launchWindowEventHandler(id, msg)
	{
		
		switch( id )
		{			
		case "login_error":
			alert( "Yahoo! Mail Beta experienced a login error: " + msg);
			cancelLaunch();
			break;

To see where this function is called from, a quick cheat might be to edit the source and call a non-existing function just inside it to check the stack trace in the error console. Then work backwards..

Actually, the patch itself came to the rescue and helped me find the error. To quickly add debug code to the patch I just used Opera's opera:config to turn off browser.js and make Opera read the browser.js file as a user JavaScript instead. When the signature check no longer applies I can modify the file:

User JavaScript can be really helpful when debugging. They partly make up the lack of a proper JavaScript debugger: you can replace specific functions with your own modified versions and do a lot of cool, custom stuff. In this case I happened to base my debug user script on browser.js.

Playing around with the "loadXML" section of the patch I noticed that DOMParser sometimes threw "Generic" errors: aha, it's Opera's way of saying "some very specific error happened when parsing that string you pretended was XML, but I can't be bothered telling you which one".

This much I knew when resuming the analysis the next day. Some searching gave me an overview of the places where Yahoo used the DOMParser to parse XML. What next?

In my experience, you can find shortcuts and resolve issues faster if you start writing tests when you start getting an overview of what a script is doing. When I think I know what's going on, I write a test that covers what I think happens – if I'm right, I've saved time by not exploring the full details. If I'm wrong, I'll have to dig around more. At this point I felt it was test case time..

I first decided to get a copy of all the "XML" Y!Mail tried parsing and compare Opera's and Firefox's output. Again I added debug code to the patch to harvest all the XML (note the postError call):

// 194334, Y!Mail faking IE's loadXML method
HTMLElement.prototype.loadXML = function (xml) { opera.postError(xml);

I copied all that XML to a single test file and checked each parse operation for three types of results: success, exception or error document.
Here is the first test case. (Slightly modified to avoid spam: I've replaced all E-mail addresses in the markup with a bogus one).

It shows that Firefox and Opera are exactly equivalent. Firefox creates "parseerror" documents where Opera throws. They agree entirely that some of the XML Opera tries to parse is invalid. So that experiement gave me no new information ๐Ÿ˜ฆ

Where exactly is the error thrown? Time to add some try…catch and lots of debug information to every place DOMParser is used. For example here:

		O.__defineGetter__("XMLDocument",function(){ opera.postError( 'in getter '+this+' '+this._XMLDocument+' '+this.innerHTML );
			if(this.tagName!="XML"){
				return;
				
			}
			if(this._XMLDocument==null){
				try{
					this._XMLDocument=(new DOMParser()).parseFromString(this.innerHTML,"text/xml");
				}catch(e){ 
					
					opera.postError( 'died in XMLDocument getter' ); 
					
				}
			}
			return this._XMLDocument;
			
		}
		);

There is a pretty interesting observation to be made from the code above: what they try to parse here is the innerHTML of an element!. That's strange.. so the element already has an HTML DOM, but they want to serialize it with .innerHTML just to immediately parse it again??

Well.. In my line of work you learn to simply stop expecting things to make sense… Expecting sense from a typical piece of the JavaScript out there is a prejudice that can only mislead you.

What next? Well, let's focus on a single piece of apparently invalid XML. Let's pick the <subjects> one. Opera and Firefox agree that what Y!Mail asked Opera to parse is invalid XML, and it is due to the ampersand that occurs here:

Leather & Tweed

. In valid XML you can't have a sole ampersand. If you need one it must be written &amp;.

So, a theory combining the information we have now: maybe Opera sometimes outputs & and not &amp; when you read innerHTML? That might cause such problems for sure! So, let's whip up another quick test case…

let's see..

Again, theory is invalidated. Firefox and Opera sing perfectly from the same sheet here.

Where does the <subjects> markup come from? That I actually knew from random source browsing, but Fiddler would have found it easily: The localised part (url will likely change) of the Y!Mail libraries contained this code

var gXmlSubjectOMatique='<xml id="subjectOMatique"><subjects><p>Welcome to SPACE X</p><p>Astonishing feats of MENTALISM!</p>...

and tons of other funny subjects you might choose for your E-mails. And – wait – the problematic ampersand is indeed written &amp; here. Even in the source Opera gets. Somewhere that amp turns into an un-escaped &. But where and how?

Back to Fiddler to work out where Y!Mail uses the variable gXmlSubjectOMatique (finally a unique variable name it makes sense to search for – thanks Yahoo). It's here:

function w9(){var O=['<div style="display:none;">',j2(gXmlBranding+gXmlSubjectOMatique+gXmlLocaleSettings)

The w9 function just returns all that markup joined up. Where is it called from?

var O=w9();document.body.insertAdjacentHTML("afterBegin",O);appOnLoadHandler()

Right, so lots of "XML" markup is added to the HTML document with insertAdjacentHTML before Y!Mail calls the appOnLoadHandler which apparently at some point will run through those XML elements, read out their .innerHTML and send it to DOMParser. So, are ampersands inserted with inserAdjacentHTML for SOME reason treated differently in Opera?
No, they aren't. Another quick test case barking up the wrong tree.

Back to the w9 function: if you notice, it actually calls another function on those three strings. A function called j2. Wonder what it does?
Here's another quick trick: rather than searching through the code for a function declaration, if a function looks like it is visible from the global scope I usually use a bookmarklet to quickly check it out. So, back to the Y!Mail window and javascript:alert(j2); – now, what's all that about??

This is very odd.

First it just decides to do nothing if a variable is true.

  if (jo.O)
    {
      return L;
    }

I'll bet that there is some sort of browser sniffing behind that if. This really smells like a workaround for a problem in a specific browser..

The actual work going on in this function is here:

  var D = new RegExp("<xml(.*?)>(.*)");
  for (var O = 0;O < T.length;O++)
    {
      T[O] = T[O].replace(D, "<xml$1>");
    }

and it does.. it does.. eh?? looks like it wraps everything inside the XML element inside a comment tag?!? Let's see if that's right..

javascript:alert(j2( gXmlSubjectOMatique ));.
THERE YOU GO! It outputs a string starting <xml><!–…[/b]! Note that comment!

Now this is beginning to feel like the last few hundred meters of a marathon. Chances are that was the remaining piece of the puzzle: do Opera and Firefox treat ampersands differently if read with .innerHTML [b]from an HTML comment section[/b]?

[ATTACH=http://files.myopera.com/hallvors/blog/282808.htm]Yes, they do![/ATTACH]

Finally, finally, we’re at the actual cause of the problem. Opera has a bug when reading [b] with innerHTML, it outputs .

This breaks Y!Mail because they take a string of perfectly fine XML, wrap it in HTML comment tags, put it inside an <XML> tag, add it inside the BODY tag with that horrendous IE thingy called insertAdjacentHTML, serialize the DOM of said tags by reading .innerHTML, strip away the comment tags again and send the string to the DOMParser.

I can think of a few simpler, safer ways to parse XML. But apparently the "keep it simple, stupid" principle does not apply when adding compatibility layers on top of existing IE-only web apps like the Yahoo mail team has done with the webmail formerly known as Oddpost. And as usual, nothing is harder than being compatible with the workarounds against other browsers' bugs ๐Ÿ˜ฆ .

Advertisements

39 thoughts on “the backtrace of an Y!Mail debug session

  1. That was a really great post. Thanks for letting us in on how you work at Opera. Nothing like debugging someone elses code. Although I seldom [read never] have to debug JS at work, I can see similarities to the way I move forward in other scripting languages when I debug an application that is new to me. Not to be rude or anything, but I was so sure I would see a Firebug screenshot somewhere in this post. But instead you gave away a couple of Opera tricks I have never thought about before.Thanks again,- ร˜ร˜ –

  2. Nice work ๐Ÿ™‚ You don't have a shortcut for `Reload from cache` (Alt+r, e.g.) ?The script formatter doesn't work here, wasn't there a trick to have Opera popup a window with the whole code formatted?

  3. Hats off to you, that was an impressive piece of work, and fascinating to watch.I'm sure a lot of people are going to be very happy when this fix is available!

  4. Very good work dude! Hopefully we will have a nicely working Yahoo Mail in Kestrel.

    This breaks Y!Mail because they take a string of perfectly fine XML, wrap it in HTML comment tags, put it inside an <XML> tag, add it inside the BODY tag with that horrendous IE thingy called insertAdjacentHTML, serialize the DOM of said tags by reading .innerHTML, strip away the comment tags again and send the string to the DOMParser.

    Too bad that the actual code used in Yahoo Mail sucks big time. Creepy.Doesn't all that code make Yahoo Mail slow?

  5. Originally posted by dantesoft:

    You don't have a shortcut for `Reload from cache` (Alt+r, e.g.) ?[/url] That's one of the first things I do in a fresh install: set Alt+Shift+F5 to Refresh Display.

  6. Doesn't all that code make Yahoo Mail slow?

    Probably. It looks like a lot of hard work compared to, say, send a JSON object with the subjectomatique data!I can sort of see why they are using SOAP and XML with the XMLHttpRequest traffic (they get the possibly convenient responseXML DOM to walk through for responses), but the stuff that is described in this post could be a lot simpler and more efficient if they just added the data to the JS without the XML. But as I said, expecting things to make sense while debugging can only mislead you.. ๐Ÿ˜‰

  7. Wow, this really is impressive work. Bringing it down to such a simple little thing… Amazing puzzle work. I'd never have come anywhere that far! And this blog post detailing your work was most delightful! Thanks ๐Ÿ™‚

  8. You don't have a shortcut for `Reload from cache` (Alt+r, e.g.) ?

    Perhaps he didn't want to mislead users into thinking there -was- a default key shortcut?Very creepy dissection Hallvord. I've never had the patience to debug a language as grouchy as JS, but I'm certainly happy someone does! :)Offhand, why use JPEGs for screenshots? Wouldn't PNGs give better results? Certainly clearer results, thought some of them -are- kind of busy, I guess.

  9. robodesign: scriptformatter.php is currently at version 0.001 or something – I have already considered putting it in some public repository and I'll probably do that at some point.

    why use JPEGs for screenshots?

    MTKnight: just a mistake, it was late night and I used IrfanView's excellent screencapture to file feature. Didn't notice that the format and the quality was a bit coarse until it was too close to midnight to bother doing them again. Sorry to make you all dizzy :p

  10. uh ! The work you just saved me Hallvord ! :DAll those workarounds are to emulate IE's xml data islands in FF. Check your mail ๐Ÿ˜‰

  11. Great post, nice to learn how you debug website offenders. I'll try to download that Fiddler thing, I've been looking for a tool like that for a long time. Thanks!

  12. Very interesting read. I don't want to take you away from your very important Opera work, but more blog entires like this would certainly please JS hacker geeks like me :PAnyway, congrats on finding the problem, it looked like quite a pickle.

  13. Very nice description of debugging. Now.. hopefully a new release of Kestrel Alpha will be released soon so I can start using it for good!

  14. Apparently Yahoo must use different sets of code on their mail servers. us.msg1.mail.yahoo.com is different from us.msg2.mail.yahoo.com. Unfortunately, the temporary test fix only works with us.msg1.mail.yahoo.com.

  15. For now, it looks like RSS scrolling using the mouse wheel is inverted.How are updates to browser.js distributed? It seems like help>check for updates says "you are using the latest version of Opera," regardless of potentially out of date browser.js. So is browser.js automatically updated in the background, or what?

  16. That's correct – Opera won't tell you whether it found a new version of browser.js or not. That's not quite ideal, but naturally very few users would understand any browser.js – related information in any case..

  17. Hallvors, Do you know when the changes you've made will start working in the later builds? Anything after build 9500 doesn't work, so I will stick with this build. ๐Ÿ™‚

  18. Yeah, the builds after 9.500 completely break Y!Mail.Also, I was just updated to version 0.7.1 (f308). Things are mostly the same as b4 in Opera, except stability and speed took a serious hit. Simple things like hitting "New Email Message" just cause it to sit there trying to load perpetually (it seriously does not work at all). The Y!Mail/Opera combo for now is just not useable.Please keep up the good work for Y!Mail/Opera compatibility. This is a very serious issue.

  19. To be honest I have done nothing on Y!Mail for a while because of some temporary crash bugs in internal builds that made testing it difficult. I think those are resolved so I'll dive right back in when a build proves stable. You're absolutely right that for now only the first alpha of 9.5 works with Y!Mail.

  20. You didn't mention that Fiddler can also be used to change the request sent to the server, and the response returned from the server.This can be useful in this kind of debugging sessions.

  21. Opera 9.5 Beta has really worked wonders as far as Y!Mail compatibility. So far all of the many Y!Mail features are working. This includes chat/IM, which has never before functioned in Opera. Everything is working quickly and properly, despite only a couple minor layout problems and occasional slowups. This came just in time as I was starting to move to IE7Pro. Now, I'm back on Opera (thankfully, cuz it's much faster).Excellent work!

  22. Anonymous writes:Yahoo Mail is not working with Opera 10. They have worked just fine in the last year until July 2009.

  23. Anonymous writes:I use the Yahoo Classic version. It works well with all browsers, except Opera (10.10). I can see my mail, but the left-side column with my folders as well as the trash, the items sent, etc. is invisible. Is there a solution or evident problem?

  24. Chris writes:Hi,Got a new problem in new yahoo mail beta and the new opera 11, it seems that the new yahoo mail beta is not working in opera 11, do you have any fix for this?Thanks!

  25. Yahoo mail worked for me in Opera 10, but now with 11 it doesn't work any more :(. The funny thing is that when I upgraded from 10 to 11 it still worked, but when I installed a new hard drive and installed Opera from scratch it didn't work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s