You might be familiar with XTabulator. It lets you sling CSV, TAB, and other tabular data files around, massaging these inherently structureless data files to your own needs. Indeed that’s the point of, say, a CSV (comma-separated value) file. In the years since I’ve released 1.0, I’ve used XTabulator extensively in my own work for a number of reasons. You could say I eat my own dog food, if you were given to tired phrases.

As I write this, I’m on the verge of releasing XTabulator 2. It almost didn’t happen.

Indeed, for the first half of 2009, another developer temporarily owned the intellectual property of the original XTabulator (and Temporis, another of my apps). This arrangement didn’t work out. I’m upset about that but not for myself or my bottom line, but rather for the unfortunate mess this creates for the customers who’ve bought these apps since they changed hands. At this moment, neither are available for sale pending some hard work and tough business decisions.

I ramble. Back to XTabulator.

It became clear that this neglected little application needed a serious update. That update would need to be free to those who got caught up in the acquisition mess and cheap to those who’ve used it for years with no real features (no, it can’t be free for all … Bartas Technologies is a business and my long hours of work need to be justified). The update needed to be a compelling and useful 2.0 rebirth. The Second Coming of XTabulator, as it were.

What’s Wrong with 1.0?

To make a “compelling and useful” update, I had to take a good hard look at the existing version. Were XTabulator not useful, I wouldn’t have made sales and I certainly wouldn’t have used it for years, dealing with its (admitted) shortcomings. So, the shortcomings.

The truth is, it took only a month to write and a week to set up for launch. Its parser “engine” is rather dumb. I dare use the politically-incorrect label “retarded”. Sure, it does what it says: it reads delimited data files. What it doesn’t do is correctly handle quoted contents (for example, a field “CITYSTATE” might contain “New Market, MD” but would be blindly interpreted as two fields in a CSV file). It belligerently ignores file encodings. It knows enough to assume the delimiter was a comma if the file’s extension was “csv” but makes you confirm it anyway. The UI freezes until it’s done reading the file – a frustrating “feature” when used with large files. Speaking of large files, it does a respectable job once the file is open but nowhere near respectable enough for my tastes when “huge” files are involved. Note I use the imprecise term “huge” because a hard metric like “100,000,000 lines” is meaningless in the face of field contents, number of fields, encoding, quoting, etc. Last but not least is that god-awful metal window. What was I thinking? Answer: “Because I can.”

In short, it is a 1.0 product and it shows.

New Hotness

Okay, so XTabulator 1 is old’n'busted. What would it take to make the new hotness? Here was a list I created when I finally decided I was going to rewrite the whole damned thing:

• Fast multi-threaded, progress-reportin’, quoted-field-caressin’, file-encoding-lovin’, parsing goodness (i.e. “better parser”).
• “Just work” file opening when possible (meaning “if the file has a known extension, just open the damned thing”).
• “Just work” file saving (meaning “if any of a column’s contents need to be quoted, quote them automatically when saved, if the extension is changed to one that is known, just convert the thing, no-questions-asked”).
• Easy, intuitive column-level actions.
• Easily-editable column headers.
• Fast and easy sorting.
• Search that makes sense (more on this in a bit).
• Arbitrary drag and drop row ordering.
• Easily-changed encodings and line endings.
• Smarter copy-and-paste (and “new document from selection”).
• Export conversion options.

Parser

I’d considered updating XTabulator a couple of years ago. I admit every time I thought about it I groaned inwardly. Mainly because writing a good CSV (I use this term generically) parser is quite a bit more involved than simply separating each line into tokens based on a comma. There are some good “lower-level” (i.e. “non-Cocoa”) examples out there but adapting them to play nicely with the native Mac OS X API (including encodings, storage, etc.) was a chore in itself.

When I decided to write version 2, I remembered a “recent” article by Drew McCormack on MacResearch.org that demonstrated a very clean solution that seamlessly handles quotes and encodings. It required far less work to adopt, all things considered, and with some not-so-simple modifications, it could be made to accept any delimiter and efficiently report parsing progress.

The final result is that XTabulator 2 is fast and efficient. When opening large files, it shows its progress (in terms of rows processed), and it automatically detects the line endings and file encoding.

A word on file size. I’ll be blunt: the capacity of XTabulator 2 is good enough. It is reasonably fast on files that are far larger than a user is likely to want to manually manipulate with a GUI application. Frankly, if you’re working on files large enough to crush XTabulator 2 on a modern Mac, you should be using some sort of script rather than a GUI app anyway.

In other words, if XTabulator is too slow for your large file, you’re doing it wrong. There. I said it.

“Just Work” Opening

I’ll say something else, too: that damned sheet that pops down when opening files in XTabulator 1 is obnoxious. After years of using it, I feel this is the thing I hate most about the application. Yes, even more than the aesthetics (or lack thereof) of the metal window. Damn do I hate it.

So here’s the deal: it’s gone. I was originally quite worried that users would need help making sure they really wanted to interpret the file with the settings XTabulator guessed. Really, they don’t. I sure as hell don’t. If I’m opening a “.csv” file and it’s not really CSV, there’s nothing XTabulator can do about it anyway and it’s really my fault. Would I change a Word document’s extension to “.txt” and try opening it with text editor?

Maybe.

If I did, it’d be my own dumb fault for expecting it to interpret the file correctly, now wouldn’t it? I specifically promised a text file (by using the “.txt” extension). Is it really every application’s job to recognize the error and say, “tsk, tsk, user, this doesn’t look like a text file”? Nope. To do it right would require unsustainable effort and, really, who is the application to say it was “wrong” to open it that way?

So following this logic XTabulator 2 treats the two common extensions (CSV – “comma-separated values”, and TAB – “tab-delimited”) as first-class documents with an expected format. If you broke your promise and the app doesn’t correctly interpret the file, then it’s your fault. Throw a fit, then deal with it appropriately.

If, on the other hand, you attempt to open a file with any other extension (like .txt), or with no extension at all, XTabulator will ask you only for the field separator (the delimiter … yes, argumentative user on MacUpdate, the delimiter, look it up). It can infer the line endings and encoding on its own. Once you provide the delimiter, XTabulator will continue on its merry way as if it knew the answer to that question all along. Right or wrong.

The drawback to incorrectly interpreting a file when it’s opened is the same as the Word-doc-in-text-editor example: if you save the file in this state, you’ve probably just killed it. Sorry about your luck, but “them’s the breaks”.

Regarding quoted fields, this works as expected when the file is opened. You’ll note a distinct lack of quotes in the editor. This isn’t a mistake. The quotes are dropped because they’re really not needed while editing, especially since the delimiter can change upon saving. More on that in a bit.

“Just Work” Saving

Speaking of saving (and assuming you’re doing so with full knowledge that it’s what you really want – really, really want – to do), the same “get-the-hell-out-of-my-way” approach is applied here as well.

First, the file is automatically saved with whatever encoding you’ve selected at the bottom of the editor. That happens to be what the operating system itself detected when it opened the file, by default. If you change it, you’re dutifully warned of the potential consequences, but when you save, that’s that. Mostly the same for line encoding (sans warning – this isn’t really a “destructive” action, any more than making a simple edit is “destructive”).

Second, back to quoted fields, quotes should be automatic. In XTabulator 2, they are. If you need precise control over quoted fields, you’re mostly out of luck. At least in 2.0. XTabulator automatically detects any fields in a column that contain the chosen delimiter (such as the “New Market, MD” example above). For simplicity, if one field in a “column” needs to be quoted, every field in that column is quoted when the file is saved. So it is written, so it is done.

Third, when using Save As, selecting a different file type just works (well, as long as it’s CSV or TAB). If the extension is anything but CSV and TAB, the delimiter is left intact. In other words, if you open a “Addresses.txt” file that is “pipe-delimited” (the | character), a Save or Save As will leave the pipe character as the delimiter, as it should. If you choose known type (CSV or TAB), the delimiter is automatically changed (to a comma or a tab, respectively). All saves will now use the new delimiter. This greatly reduces the complexity and the need for any real thought. Converting between common formats is drop-dead-simple.

Hot Column-on-Column Action

Through years of use and customer feedback I came to realize that, beyond adding and removing rows and editing field contents, most editing actions were on a per-column basis. I hadn’t considered this in the design of version 1. In fact, I practically viewed the entire application design through “row-colored glasses” to use a very witty pun.

In version 2, not only do rows and columns get equal billing in terms of design consideration, but columns are taken one step further. Deep consideration was given to the conspicuous absence of controls for directly manipulating the columns themselves. Why select a column then go traipsing through the menu system for the right “columney” command or a clutter of toolbar buttons? The trouble is, the standard Mac table control behavior doesn’t think columns are as important as I do. I call shenanigans!

XTabulator casts off the constricting mores of the “standard table control”. Sorting by a column should be easy, and with a standard table it is (that’s good). However, selecting a column also automatically sorts the table by that column (that’s bad). The header title is also not editable at all. Design agony!

The solution to this problem is (for now) unique to XTabulator. Clicking a column only selects it (but that’s not the unique part). Hovering over a column reveals a down-arrow-button on the right edge of the column header. Clicking it reveals a plethora of column actions (including sorting) which are also found in the main menu. These actions pertain to the column to which the menu and button belong. Double-clicking the column header reveals an editor “bubble” in which you can edit the column’s title and press return (or click “done”) to save the change or escape (or click outside) to cancel the edit. Dragging a column works the same.

The benefits are clear – direct “work on what you clicked” action, but the drawback is the loss of single-click-to-select-and-sort. Small change. Although far from perfect, the solution solves all the problems without creating any big problems in return.

Maybe inline title editing (directly in the header) will make an appearance in the future, but for now the text area is too small to make a good-looking (i.e., a legible size for those with less-than-perfect-vision) inline editor.

Some Sorta’ Sortin’

In addition to the column actions above, I wanted to expand on the sorting just a bit. Two big complaints about version 1 were the lack of arbitrary row ordering (i.e., dragging rows around into any order) and stupid (blind alphabetical) sorting.

Dragging rows around: check. Sorting? There’s a catch. XTabulator will no longer keep rows sorted. Sorting is now a one-off column action. If you sort by a “last name” column, then insert a “Z” name in the middle of the rows, that’s where it stays … unless you once again sort by last name. The benefit is, rows always stay where you put them.

Another benefit is that it no longer blindly sorts alphabetically. In this regard, XTabulator behaves more like the Finder. That is, numbers are treated as numbers, when possible. No more “1, 10, 11, 2, 3, 4…”.

What if you want to be able to sort (to easily find things) but restore the original order? It’s easy enough to add a column (say, “Original Order”), then use a handy Fill column action – no, not a “Phil Collins handy action” … yeash – to fill the column with ascending numbers. Do this first, sort to your heart’s content, then, when you’re ready to save, sort by the Original Order column and remove it. While a bit fiddly, it works well.

Searching

Speaking of finding things (no, no cutesy subheading this time, sorry), search has been given a solid once-over and then some. The trend these days is to have a “slide-down search bar” that appears when you hit Cmd-F. Fine.

XTabulator 2 has this. It also sports additional navigation and selection controls. The back and forward arrows jump between previous and next matches (handy for thousands of rows) while the middle button selects all matches (all are selected when first found by default). This makes it very easy to search for all “XTabulator” orders in sales records, for example, then with a single keyboard shortcut, create a new document containing only those records (complete with columns and headers).

Another control in the search bar allows you to fade out non-matching rows. This makes search matches far easier to spot at a glance (especially while scrolling). Why go to such trouble? Well, here we run afoul of another design conundrum.

XTabulator once again finds itself at odds with standard UI when searching. While most apps “filter out” non-matches, this poses a user interaction problem that is particularly painful in an application like XTabulator. What happens if you sort or manually reorder search results? The third search result row is not necessarily the third row in the whole file. In this case, sorting and reordering makes no sense.

The solution is to either disallow reordering and sorting while in search (boo) or simply not to filter the rows (yay?). I chose the latter. To make this more user-friendly, search result navigation and easy visual identification was added.

The End(ings Control) is Near (and So is the Encoding Control)

Both the encoding and line endings controls are right on the bottom bar of the document. This deserved its own subheading and paragraph. That is all.

Copy & Paste & New Document From Selection, Too

Copy and Paste works more like users expect in version 2. You can cut, copy, and paste rows or columns. Do it within a document. Do it between documents. Do it from XTabulator into another app (the result is the expected line for a row and multi-line text for a column). You can even paste into Excel in the case of a tab-separated document (Excel does the right thing with tab-separated text in the pasteboard).

Further, as I mentioned before, you can select one or more rows in one document and, with a keyboard shortcut or menu command, you can create a new document with those rows. XTabulator even preserves the columns and their headers (titles) in the new document. Again, a few swipes of the keyboard and you can find rows and isolate them in their very own document for convenience or the safety of your original document. This was a must-have for my own needs.

Export Ban Lifted

Version 1 only dealt in tabular data. Opening and saving. That was it. Version 2 goes a step further and offers export (really, “conversion”) to three oft-requested formats: HTML table, XML (PLIST format), and Sqlite database. More formats may be forthcoming.

I call it “export” because it’s definitely a conversion to a different format and “export” is the more common word for “convert” in Mac parlance. No word (or promises) on “import”.

Other Niceties

XTabulator 2 uses Sparkle for in-place updating, has (or will have – it’s only partially finished) a nicer, more detailed help book, and is quite a bit more responsive on the whole.

The Not-So-Niceties

With everything there is a catch. XTabulator 2 is a paid upgrade (with the exceptions noted in the beginning of this very long diatribe). It requires Mac OS X 10.5 or above (yes, really … it made a lot of things far easier to develop, so much so that 10.5 is the only reason some of these things were developed at all).

Finally, the restrictions for unregistered users. Be warned (actually, you’ll “be warned” by the app itself every time) that XTabulator 2 will insert a record with “XTabulator is Unregistered” in each field when you save or export. It’s easy enough to post-process and doesn’t interfere with your evaluation. Additionally, you’ll be “reminded” (I believe the common term is “nagged”) to register every twenty minutes. If you’re using it enough that a reminder every twenty minutes happens enough to bother you, you’re not “evaluating” it any more, you’re just using it.

In Closing

A big, heart-felt thank you to my customers. With few exceptions (some deserved, some ridiculous), you’ve recognized good work and rewarded it by supporting the business (and the person) behind it. That you’re reading this demonstrates you’re interested not only in the product but in the effort, process, and person behind it. Building and maintaining a useful software application is hard, brain-melting work but it’s work I deeply enjoy. It’s difficult to put your work out there for the entire world to see (and anonymously review) but your support makes it possible for me to make a living doing what I love and your words of encouragement gives me the strength and conviction to continue working hard to do the best I can.

Thanks so much.