Bill Bumgarner has posted a few great articles on Objective-C method tracing using dtrace:

• Objective-C: Printing Class Name from Dtrace
• Objective-C: Using Instruments to trace messages-to-nil

I have a internal tool at Omni that traces message sends by creating new method IMPs to wrap the existing ones in a trampoline, but it requires some nasty assembly stubs and has only ever worked on Sparc, ppc and x86. It is very fast, but hard to maintain and a little fragile (and doesn’t handle the nil case). A 90% solution can be obtained by extending Bill and other’s work.

Ken sent out a partial tracing solution this morning and finally got me off my keister. Here is a D script that catches ObjC message sends and dumps a OPML fragment (which can then be wrapped in the XML goo via an external shell script or whatever) and then viewed in OmniOutliner.

Download: objc-trace-opml.d.gz

This isn’t a perfect solution by any means. Some of the issues:

• This has to be run in an essentially single-threaded program; any background threads must be totally quiescent or you’ll get a mixture of output from multiple threads. It wouldn’t be too hard to modify this script to include a thread identifier or maybe just filter out anything from background threads.
• This won’t handle exceptions correctly; they’ll break the OPML nesting. Our internal tool handles this, but since NSError arrived on the scene, this is less of an issue. Just avoid tracing code with exceptions.
• dtrace is slow when matching all entries in the objc provider. The startup time on this script is something like 40 seconds on my machine. Annoying, but not the worst thing ever, since you are likely to trace once and then spend a fair bit of time examining the results.
• This particular variant of the script will only work on x86 due to the hacky way I check for objc_msgSend_stret.
• dtrace can overflow its buffer and drop events when there are massive numbers of probe hits. Run with a big buffer and trace only the exact set of methods you need. Ideally you’ll be running in the debugger, stop right before where you need to trace turn on tracing and then ‘next’ over one or two lines of code. The ‘-b’ flag to dtrace can also be used to increase the buffer size.

Some of the nice bits:

• No assembly-fu required.
• Call hierarchy is preserved and can be expanded/hoisted and such in OmniOutliner.
• Both the class of the receiver and the class of the method implementation are included. So, you can see that +[MySuperclass initialize] is getting called on MySubcass, for example.
• All method invocations are shown, so you can see nested calls to super instead of just the initial call to objc_msgSend. I’ve not tested whether cached method IMPs get traced too; normal code should get traced as expected. Swizzling, IMP caching or dynamically registering methods will possibly not work as well.
• The pointer value of the receiver is emitted. By the time you examine the output, it might be dead and gone, but a surprising amount of the time it isn’t. For example, things like static NSStrings, NSScriptClassDescriptions, interface widgets and other cache-once data may still be around for submitting to ‘po’ in the debugger.

We finally have space to grow, now that we’ve moved to our new office! There is no specific position to fill, but rather we are content to wait until we find someone who will fit well into our team. Take a look at what working at Omni is like and the general description of who we are seeking to see if this is the right place for you!

I failed to find any way to do this via the internets; but gdb, and some kindly soul inside Apple, met the challenge:

(gdb) b -[NSUserDefaults objectForKey:]
Breakpoint 17 at 0x94172fa4
(gdb) c
Continuing.
... invoke service, hit breakpoint ...
(gdb) po *(id *)($fp + 16)
NSShouldActivateForServiceRequest
(gdb)

Is there a better way to do this? Google and mdls don’t seem to know about NSShouldActivateForServiceRequest at all, so I’m guessing not.

Terminal in 10.5 has seen some great scripting improvements to match its additional functionality. Occasionally I have several Terminal sessions with some ssh‘d remote servers and then I forget which Terminals are local and which are remote.

Here is a simple little command-line AppleScript that will let you set an individual Terminal’s style from the command line; SetTerminalStyle.zip

Put the script in your PATH somewhere and then add a ssh function in your .zshrc, or equivalent:

# Show remote Terminal sessions with a different style.
# TERM_PROGRAM doesn't get set if we are logged in remotely,
# but then the originating Terminal could have done this (hopefully).
if [ "$TERM_PROGRAM" = "Apple_Terminal" ]; then
  function ssh {
    SetTerminalStyle ssh
    /usr/bin/ssh "$@"
    SetTerminalStyle default
  }
fi

Finally, create a ssh Settings entry in Terminal’s preferences:

Other obvious commands for this include sudo and su.Using RPROMPT and Terminal’s processes scripting property on tab it might be possible to get this to work without the function wrappers, but executing the AppleScript on every command was a little too slow for my taste.

Today’s post is the first in an ongoing series I’m calling OmniFocus: What We’ve Learned So Far (or OF: WWLSF, if you prefer acronyms). As we move slowly but steadily towards a feature freeze and public beta, I thought it would be interesting to get some input from various people here at Omni on things that have gone well, as well as things that have sucked challenges we didn’t anticipate—basically, the ups and downs behind building a piece of commercial software.

We’re going to start out in the technical arena, so I apologize if code-talk makes you yawn so hard you accidentally drool a little. Here is Omni’s engineering perspective on an important lesson learned during OmniFocus’s development process, which can be boiled down to: we ♥ CoreData, but not as a primary file format. 

With more on this subject, here is Tim Wood, hater of Aeron chairs, terror of the Unreal Tournament battlefield, and OmniFocus team lead:

There are many things that are great about CoreData, but using CoreData as a user-visible file format was really painful. Since inception, our xcdatamodel file has had 92 revisions, with most of those exposed to several thousand people via our automated builds. Most of these changes aren’t things that users would notice; we often add or remove precalculated summaries, denormalize data or generally change the underlying CoreData representation to make our app easier to implement and tune. Yet, with CoreData, the SQLite mapping would be busted beyond hope by adding or removing a column.

Manually building code to migrate between model versions is really not an option. If CoreData had a Rails-like migration facility where columns could be added and removed trivially via SQL ALTER statements, it might be feasible, but it still wouldn’t be good. CoreData explicitly disclaims any support for direct access to the various stores, so it isn’t a public file format and hinders our users from easy access to their data. In practical terms, we all know that a liberal application of the letter ‘Z’ will get you most of the way to accessing your data. Still, this isn’t ideal.

What CoreData is great for is building an optimized cache of your data, fetching against it and then binding it to your interface.

A couple of other key observations are that we already needed a public file format for Export (we chose a custom XML grammar, but that’s merely a detail). And, using a variant of the public file format for the pasteboard format is a great way avoid writing and testing more code (as is using your pasteboard archive/unarchive code to implement your AppleScript ‘duplicate’ support…)

Given that, I tweaked our XML archiving to support writing a set of CoreData inserts, updates and deletes as a transaction. We can then write out a small fragment of our content in a new gzipped XML file inside our document wrapper. The structure of our XML transactions is very simple with a key feature being that we can trivially merge a big batch of transaction into a single XML document that contains only the final set of objects as inserts.

On startup of OmniFocus, it scans the transaction log in the user’s document and builds a cache validation dictionary that contains:

• Version of Mac OS X
• CoreData’s version
• SVN revision of the application
• The last transaction identifier

We then open up the CoreData SQLite persistent store and peek at its metadata. If it isn’t an exact match, we close up the persistent store, and rebuild the entire thing by importing our coalesced transaction log in exactly the same way we would import one of our backup files.

There are many extra implementation details (locking, catching the insert/update/delete notification, undo/redo vs. AppleScript, two-phase commit between the XML and SQLite, …), but we are really happy with the central approach.

Some of the fun things this gives us:

• You can run the same build of the application on 10.4 and 10.5, switching regularly and not worry if CoreData is going to ignite your SQLite store.
• You can run multiple builds of OmniFocus on the same data and not lose anything (more work may be needed for major file format upgrades if there ever is one).
• If we do screw up one of our automated builds and mess up cache updating code, the user’s data doesn’t get touched and it’s just fine on the next build.
• Until the transaction log is compacted, we actually have the full record of edits and we could hypothetically implement persistent undo, allowing the user to rollback to yesterday’s version…
• … or calculate the changes they’ve made since some point in time.

The last point is really interesting and I’m hoping to make good use of that in the future for things like computer-to-computer synchronization (no, I’m not promising anything)!

For those who might be interested, Ken recently answered some interview questions over at MacApper. Topics include Ken’s Old Skool Programming Street Cred (respectably nerdy!), his thoughts on being Omni’s CEO, and upcoming development plans.

Check it out when you get a chance!

For many years now we’ve had an integrated crash reporting system. This has helped improve the stability of our applications immensely (often report now start out with “Wow, this is the first crash I’ve seen…”). But, it hasn’t always been clear (especially in the alpha or beta timeframe) how well we were doing on overall stability. We could guess by counting the number of crash reports vs. an estimate of the number of active users, but that wasn’t very convincing.

Near the beginning of June, I added some support to our software update and crash logging frameworks to keep track of things like:

• total times the application has been launched
• total number of crashes
• total amount of time the application has been running

(As always, our software update system reports its information without including any personal details, and can be disabled entirely if so desired.)

Using this, we can now chart the total number of hours OmniFocus has been running vs. the total number of crashes (reported or not!). As the pool of people testing OmniFocus goes up, or some testers go idle, or some user with large number of crashes isn’t reporting them, we don’t have to wonder as much how that affects our average crash rate.

After my latest crash fix, our rate has improved to about 8000 hours per crash. We aren’t sure yet what constitutes a reasonable lower limit for hours/crash, but this does let us notice when a fix we’ve made actually is addressing the issue. We aren’t yet tracking the number of hours that the application is active (an hour spent hidden counts the same as an hour spent in full use). Whether this matters, when averaged across a large number of users, is open to question.

Still, there are only 8760 hours in a year, so if we can get above that, we’ll be feeling pretty good.

Wim came up with a neat trick a while back that we’ve used to find and fix several bugs in our software, and to file a bunch of Radars. There are several messenger dispatch functions in the Objective-C runtime. Of particular interest here are objc_msgSend_fpret and objc_msgSend_stret. These are used by the compiler when calling a method that returns a float or struct, respectively.

Depending on your architecture, the result of such a message can be undefined when sent to nil. Messaging nil is very useful most of the time, but you can introduce rarely manifesting bugs in this case.

Looking at the disassembly for these two functions in gdb, though, gives us an easy way to catch them. Under 10.4.8/x86, we see the following:

(gdb) x/50i objc_msgSend_fpret
0x90a573c0 :        mov    4(%esp),%eax
0x90a573c4 :      test   %eax,%eax
0x90a573c6 :      je     0x90a57420
0x90a573c8 :      mov    0(%eax),%eax
...

That is, load the first argument, check for zero, if so jump to 0x90a57420.

Likewise, in objc_msgSend_stret:

(gdb) x/50i objc_msgSend_stret
0x90a57340 :        mov    8(%esp),%eax
0x90a57344 :      test   %eax,%eax
0x90a57346 :      je     0x90a573a0
0x90a57348 :      mov    0(%eax),%eax
...

In our ~/.gdbinit we can have:

# Nil-handling path for objc_msgSend_fpret.
b *0x90a57420
comm
  p (char *)$ecx
end
# Nil-handling path for objc_msgSend_stret.
b *0x90a573a0
comm
  p (char *)$ecx
end

(where the print command shows the selector).

Omni’s currently looking for a coder to help us implement our award winning apps. Send in a resumé and join the fun!

Welcome to the third, and for this week anyway, final installment of “You Ask, We Answer!”. Brought to you by Diet Coke and the letter Q.

(Q for Qwality!)

Ayjay asked, many Mac developers have moved away from the use of drawers (especially now that Apple has taken them away from Mail) but you guys still feature drawers-a-plenty. What do you like about drawers? Have you thought about any other ways of implementing the functionality that drawers give you?

Ooh, good question. I had to call in the troops for help on this one, since my opinions on drawers are mainly limited to the kind you put your socks in. Ken, our CEO, and Bill, our UI Lead, put their heads together to answer you:

‘We like drawers because they are a great place for content that belongs to the main window but doesn’t necessarily need to be there all the time. They’re great for “source lists”, from which you can choose what to view in the main window, like OmniGraffle’s canvases or OmniWeb’s tabs. Perhaps best of all, unlike a sidebar, you can show, hide, or resize a drawer all day long without affecting the dimensions or layout of the main content. And in Omni apps, you can move the drawer to whichever side of the window you prefer by Option-clicking the drawer button in the toolbar.

The problem with drawers, of course, is that the things just don’t look modern. While the rest of Mac OS X interface was getting the sleek plastic or metal treatment, drawers are still as pinstripey, space-wastey, and noisy as they were the day they were introduced. Unless we want to cobble together and maintain some sort of custom fake drawer ourselves (or—gasp!—Apple actually updates drawers’ appearance), we’re going to have to get away from drawers eventually.

In meetings for our new products, we’ve talked about how to deal with this problem, and we think we may have come up with a good hybrid of the useful drawer and the sleek sidebar. You may end up seeing the first incarnation of it in OmniFocus, if we can do it right.’

Spencer says, I have had no luck at all with storing OmniGraffle documents in Subverson. The icon seems to contain an illegal character or something.

Here’s the response from our OmniGraffle tech support/product manager NINJA EXTRAORDINAIRE:

‘This is likely due to OmniGraffle saving the files out as packages, which other file systems can have difficulty dealing with.

OmniGraffle will automatically save a file out to a package if an image or external graphic is present in the document; there is a hidden preference to avoid this behavior that can be enabled from the command line. To get OmniGraffle to always save as a “flat” file (which will have no problems on other filesystems), open up Terminal.app and paste this in:

defaults write com.omnigroup.OmniGraffle PrivateGraffleFlatFile 0

Afterwards, new documents should always save as a monolithic file, you may have to perform a “Save As” for existing documents to convert them.’

Man, I’m loving this whole ‘fob off the hard questions on other Omni employees’ gig. What else have you got, commenters? Bring. It. On! *spirit hands*

Matt wants to know, Do you guys plan on fixing RSS anytime soon? Its a sometimes it work, sometimes it doesn’t work symptom. Usually I have to relaunch OmniWeb to get it to recheck RSS feeds- it doesn’t do it by itself even though I have it set to recheck the feeds every hour.

Dang, this one’s less fun to answer. Turns out we’ve seen this issue and we’re able to reproduce it. It’s a bug that we’re hoping to fix in an upcoming release, after the 5.5.1 update. Sorry about that!

Conor asks, Can you tell me if OmniFocus will liason with OmniPlan so that you can plan projects in OmniPlan and then download your personal tasks into OmniFocus? Also, are there any plans to allow Wintel users to edit OmniPlan? I work in a mixed-platform office and, while I do most of the project planning on my mac, it would be nice to enable other employees to check off tasks, etc.

We would love for OmniFocus and OmniPlan to work together that way, but I think it’s safe to say that they won’t for OmniFocus 1.0. We’re trying to limit the scope to what we can actually get out the door in a (hopefully) reasonable amount of time, but it’s definitely on the plate for future consideration.

As for Wintel users…well, we likely won’t ever have a cross-platform version of OmniPlan, but you can use OmniPlan to export to .mpx, .mpp, and MSPDI .xml for sharing with Microsoft Project and other project management applications. You can also export to a .csv file for import into Excel, and if people just need to see the data, not update it, you can export the Gantt, outline or both into several different image formats (PDF, PNG, TIFF, JPEG). And! You can export to html – either a single table of tasks, or a mini-website with a Gantt chart, tables, and calendar files that can be imported into iCal, Outlook or other calendaring apps.

Thanks for all your questions, folks, and if I didn’t get to yours this week, my apologies. Please stay tuned for an Exciting Announcement (note: your excitement may vary) about OmniWeb I will be posting later today. Same blog time, same blog channel.