Frustrations with Gmail

Not every relationship is all rosy…

(man, I never thought I’d say this, but the on-screen keyboard is actually faster and easier to use than the hardware keyboard!!!)

Anyhow… when I switched to Android I switched my email notifications from SMS to Gmail. A little background:

I run my own personal email server, and while it supports IMAP Push, I’ve never found a client implementation of that protocol that had acceptable battery usage. Regardless, while it’s nice to get your emails immediately and be able to perform some action on them immediately, I’ve always wanted to maintain strict control over which emails generate a “new mail” alert.

There are a number of ways around that triage problem. But then you get into complexities over how you want your mail organized vs. what you want to be notified about… Long story short, I opted to create an out-of-band notification system specifically for the emails I was interested in. This has the added bonus of being extensible, so that I can define times of day that notifications are active, complex state tables that affect notification eligibility, and things like remote awareness.

Anyhow, these out-of-band notifications were traditionally delivered via SMS, and in the WinMo world I had coded a solution whereby a second, specially-formatted SMS would tell Pocket Outlook to sync email. The result was a poor-man’s push solution – perfectly functional, if not the most efficient.

So that’s the background. Since I place great value on the flexibility of this notification system, it was paramount that I could carry over its core intent to my Android smartphone. And I opted to ditch SMS and go with Gmail as the delivery mechanism.

So that’s the background then.

What I’ve been finding, much to my dismay, is that I’ve been missing notifications. In the SMS world it was possible that a notification would get delayed, or lost altogether – but I didn’t expect that from Gmail’s push notification system running on Android 2.2.

After some digging I came to realize that quite a few of these missed notifications were indeed making it to my phone, but they were ending up in the trash and hence no alert was being generated by the Gmail app.

Gosh darnit! Chock it up to Gmail’s “conversatons”, where your emails are grouped (ie., “threaded”) by subject. So if you decide to delete a single message in a conversation (which is something you can’t do in the mobile app – more on that in a minute) then the whole conversation would have a “trash” label applied to it.

And it seems that when a conversation has the trash label applied, the mobile Gmail app removes it from the inbox view. Critically, it won’t alert because the new message is part of a “deleted” conversation.

Now, I haven’t tested this, but it’s possible that things would work differently if you delete a single message in the Gmail web app. But on the Android app no such option exists – it’s the conversation or nothing at all.

Now I have to imagine that this is of the utmost frustration to people who use Gmail (and the Gmail mobile app) as their primary mail client. Fortunately I’m not one of those people, so I’m testing a solution (and it may be that this can apply to the aforementioned group of people too – unless they’ve figured it out already).

Basically, you don’t delete anything.

This is something that Gmail originally trumpeted – that you now had so much space that you’d never to delete an email again. And because you need some way to manage a huge inbox with email that’s never deleted, and people generally pout if they can’t permanentky delete their own email, Google acquiesced and added some trash “label” and other what-not to appease the masses.

But in my case those solutions don’t work. I can get by without deleting because I only care about new emails – or more accurately, new notifications of new emails.

So my solution is not to delete, but also, to only sync 1 day worth of email out of my inbox. That way, the old undeleted stuff will fall out of view in the mobile Gmail app. I’m also syncing “all” starred items, so if something is of interest and I need it to stick around for a few days – like a URL that I’ve emailed myself – then I can star it and it will remain in view.

That’s the theory. I’ve only just now made all the changes, so you’ll need to follow tradition and give me a few days to deal with the (inevitable?) fallout.

Farewell Latitude (for now) (update: it’s back)

So that’s it then – no more Google Latitude on my phone or the wifey’s.

It’s not that Latitude has lost its usefulness.  Far from it – it’s quite handy.  Rather, it has lost its practicality.

For whatever reason, Google took it upon itself to change the update frequency of Latitude, and this has had a noticeably-adverse affect on battery life.  Google’s own search engine will yield results explaining that the frequency went from something like 42/1 to 6/2 – ie, a 42 minute period when not moving and 1 minute when moving, to 6 minutes when not moving and 2 minutes when moving.

Now I don’t know if these numbers are correct, but I do know that something has definitely been amiss in the Latitude department for the past month or two.  It started when Shelly’s S60v3-based phone would no longer post Latitude updates.  Google seemed to have resolved this issue about a week ago, but in looking into the problem I came across other posts talking about increased battery usage, attributing said increases to the change in update/polling period.  So once functionality was restored on Shelly’s phone, I noted that she seemed to have gone from two or three days between requiring a charge, to needing to charge each night.  Hrmm.

So the first phone to have Latitude disabled was hers.  And wouldn’t you know it, battery life was magically restored.  And while I’ve always known that Latitude used some of our precious mobile battery life, I certainly couldn’t justify a 33-50% reduction in said life.

Battery life on my own phone is harder to characterize, as I tend do Do More Things(tm) on it, and I also tend to update my apps regularly.  Nonetheless, even with charging my phone in the car for one or both of the daily commutes, I was still getting alerts every night warning of 10-15% battery remaining – and I distinctly recall the ability to do a good amount of web browsing without any battery concerns in days past.

So today is the first day that I’ve disabled Latitude on my phone as well, and I have no intention of charging it in the car today either.  We’ll see what happens this evening.

It’s unfortunate really, because I’ve found that Latitude is one of those things where you like to Fire and Forget(tm) – that is, turn on the tracking service (privacy concerns notwithstanding) and it’s always available when you need it.  Sometimes it’s a safety consideration, other times it’s a simple matter of convenience.  I’ve even taken to using the Location History to do some data mining.

But until the battery life issues are worked out, it’s no more always-on Latitude for me.

[update 2011/04/06] Well Latitude is back on my phone.  As usual a number of variables have changed on my phone – WiFi is being turned off by Tasker unless I’m on a “home” cell tower, and I’ve upgraded Maps to the latest and greatest (v5.2 I think).

Anyhow, battery life is acceptable at this point (although a Mugen 1800mAh battery is en-route).  And it would seem that updates aren’t going out every 6 minutes – on my phone at least.  Looking through the data logs on Shelly’s phone I was dismayed to see packet data activity every 6 minutes throughout the day – surely an indication of Latitude phoning home.

You may wonder why Latitude is again active on the wifey’s phone.  Hey, you’ll have to ask her – I turned it off a few times and it’s magically gone back on again each time.

So I’ll have to take a look into the WiFi settings on her phone.  I was under the impression that Symbian is very sane about powering the WiFi module, only doing so when an application explicitly requests data access.  And while this could still allow Latitude to request WiFi when it wakes up every 6 minutes, I was of the belief that the user had to explicitely select a predefined WiFi “access point” to use, or else tell the phone to go searching for an available WiFi network.  Which is to say, unless their’s human intervention, the phone shouldn’t be turning the WiFi module on.

So either my understanding of WiFi on Symbian is wrong, or Latitude on Symbian is a just a different beast altogether and it’s misbehaving badly.

More location awareness

Having gotten my feet wet with occupant detection for the home surveillance system, I recently decided to tackle a similar scenario for another application.

This one was bought to the fore courtesy of two seemingly unrelated occurances.  First, was the move to Android from Windows Mobile.  My Touch Pro had a handy menu setup whereby I could easily send automated messages to my home network, and have it Do Things(tm) – notably, whether to notify me on my phone of received work emails.  So far I’ve found no easy solution to do this in Android, save the laborious method of writing and sending the actual emails which command my home network.

But I was willing to let it slide.  Until the second occurence – an apparent increase in the frequency of impromptu meetings at work.  With the need to be away from my desk, but still being at work, I would find myself often penning the emails to my home system telling it to direct work emails to my phone.

The obvious question here is – why not send work emails to my phone all the time?  And the answer should be equally obvious to the (non-existent) masses who read my blog missives; basically, I prefer to exercise a modicum of control over a series of repeating notifications for things that may not be of interest to me at the time.  For personal emails there’s a set of rules that determine what fires a notification and what gets summarized at some point in the future.  For work, I don’t see a point in being notified – by my phone – of a new email if I’m sitting at my desk staring at Outlook.

My excursions into the Bluetooth-powered occupant awareness solution at home have yielded favourable results, and ever since then I’ve sought to automate more tasks.  Many years ago I had written a Win32 program that interfaced with my personal mail server at home, displaying new emails as one-liners in a window that lived in the system tray icon.  The program would determine if I was at my desk based on the screensaver; an active screensaver meant that I was away, and no screensaver meant I was there – much the same way that IM applications determine your away status.  I decided that something similar was required for this new project.

What I ended up with was a combination of a Win32 console application and a WinLogon Notification DLL.  The Win32 console application can poll for the state of the screensaver, but the DLL receives events every time the screensaver starts/stops, or a logon/logoff occurs, or a the workstation is locked/unlocked.  Obviously the DLL has a richer set of conditions available to it than the console app, plus there’s definitely something to be said for event-driven processes vs. polling processes.

But both have their place.  In my implementation, the console app sits on a socket waiting for somebody to connect and tell it that my away state has changed.  This can be the DLL, or it can be a polling thread which is launched by the console app itself – the thread wakes up every x minutes, checks the screensaver, and also pings the main thread with the current state.  The main thread then sends an HTTP request to my home server, which then does all the work required to setup my mail server to notify my phone (or not to notify my phone) when a work email is received.

So long story short – it works.  I’m sure there will be some bugs to work out, but I’m just loving the feeling of having busy little tech bees working behind the scenes, Making Things Happen(tm), without any direction on my part. <g>

Trust (Me)

I’m no psychologist, and most of these Thoughts are based on my own experiences, or my own observations, or some realization that has suddenly made itself apparent – welcome or not – and perhaps a little later than it has to most other people.

Sometimes I’m just Thinking out loud, asking myself (and by extension, asking you) questions in an attempt to gauge my own wisdom – or more accurately, lack thereof.

So on the way to work this morning I started thinking about the notion of Trust.  Admittedly that’s quite a large concept to tackle, but then I got to thinking specifically about something a little easer to handle – what it means when somebody says “Trust me“.

First and foremost: nobody is perfect.  Everybody makes mistakes.  Children may not realize this; they’re too busy Getting On With It(tm), like getting up to run again after they’ve tripped over their own foot (=mistake).  Or crying to seek attention after being admonished for hitting another child (=mistake).  Or squeezing by after Daddy backed into them in the kitchen because Daddy didn’t see them (=Daddy’s mistake).

No, I think that children don’t truly understand what a mistake is until they learn what consequences are, and that there’s something called “Cause and Effect” which links mistakes to consequences.  It’s only after that, I think, that they can start to evaluate their own actions (mistakes), and other people’s actions (their mistakes) and think about what may come next if those actions occur (the consequences).

But I digress.

Adults, at the very least, should know they can make mistakes and that others can make mistakes.  Hopefully they can also admit when they’ve made a mistake.  And – hopefully again – they can understand when somebody else admits to having made a mistake.

So back to Trust (Me).  A classic and well-known trust game has you turn around so that your back faces somebody, then you lean back and hope that the person will catch you.  And assuming that you decide to put your fate in this person’s hands, one of two things can happen next:

  1. The person catches you.
  2. The person does not catch you.

The question is – if the person does not catch you, is your Trust in that person shattered?

The question I asked myself on the way to work this morning was this: was I trusting that I would be caught, or that the person would try their best to catch me?

At that point it dawned on me – and you may disagree – that often times we place our trust in the “thing” rather than placing our trust in the person.  Christians may be used to trusting in God that things will work out, even if the things don’t work out the way we expect them to.  But how often do we extend that same consideration to the people in our lives who have made the same promises to Love, Honour and Protect us?

People who we believe to be responsible.  People who have demonstrated that they have our best interests at heart.  People who seem to be diligent in their efforts.

Children trust their parents implicitly, and forgive them in no time at all.  It’s only as we get older that we start to understand what a mistake is and what the consequences are, and then we focus less on the person and more on the particular outcome – or possible outcome.  Our trust wavers, when in fact our relationship hasn’t changed – only our focus has, on the thing rather than the person.  For some reason we expect perfection, knowing full well that there’s no such thing.

And while I can admit that some things border on unforgiveable – infidelity, crime, lies – I also submit that Trust issues impact us in matters that are far less obvious and consequential yet far more frequent.  Things that seem trivial – like the infamous “backseat driver” – but are serious nonetheless because they speak to the issue of Trust.

It would seem, then, that the question of trusting somebody should be a last resort.  And as is the case with most issues of Life as we age, what was once an implicit thing now requires a decision and concerted effort on our part.  We must make the decision to relegate that question to the back of the line; behind things like practicality (“were my expectations fair?”), relevance (“does it need to be corrected?”), remorse (“is s/he making an effort to correct it?”), and mutual ownership (“how can we correct this?”).

It’s easy to feel betrayed when you lean back and your bum touches the ground.  But then you realize that it’s a Good Thing(tm) to have somebody there who’ll do their best to keep you standing.

Is he smarter than the old man already?

So it’s been interesting putting Andrew to bed lately. Experiences have ranged from incessant screaming – usually as a result of falling asleep in the car and being rudely awakened upon getting home – to a general displeasure with the notion of being left alone to go to bed, to being quite accommodating and laying down when asked.

Being the male parental unit, I have the natural (dis)pleasure of having no natural maternal instinct with which to justify my actions and decisions. Which is a long way of saying that the Mrs. does most of the intricate caretaking, which seems to include bedtime.

But for whatever reason, the little man has been giving Mommy a hard time of late in matters of nocturnal slumber – while I seem to get his best behaviour, which includes laying down upon command and remaining generally content until Sleep finally comes knocking.

Mommy shouldn’t take it personally. Kids can be fickle, especially at the tender age of Almost-2.

But the little guy definitely had me on pins and needles this evening, as he seemed reluctant to lay his head on his little pillow despite my fervent requests. This evening he had elected to be partial to a toy airplane that Mommy had brought home a few weeks ago, and in an effort to keep the boat stable we tend to let him hold on to his favoured toy after bathing and just before bedtime. Normally he’s fine giving it up just before he goes in the crib, but tonight the signals seemed muted.

So there we were – me with blanket in hand and he sitting in the crib with plane held high. “Oh no” I thought; “He won’t be so cooperative tonight”. And for what seemed like an eternity, there he sat, plane held high, with me holding his blanket waiting for him to lay down so I could cover him.

Then my heart leapt into my throat as he stood up; surely this was the beginning of an hour-long protest. My ability to bid him a happy goodnight had apparently waned.

And then, he raised the plane even higher and moved it convincingly in my direction. And finally I understood – it was bedtime Daddy, not playtime. The plane had no business being in bed with him, and he wanted me to take it. And I having done as he wanted, he immediately lay down to begin his customary soft soliloquy, culminating 20 minutes later – as usual – with Sleep.

Thanks son 🙂

Using up/down arrows with the script.aculo.us Autocompleter

I came across a problem recently, using the Autocompleter API of the script.aculo.us library.  Some amount of Googling later and I still couldn’t find a solution, but I discovered that there was misery in company – other devs were having the same issue.

Basically, when the autocomplete list is generated, and said list has a scrollbar on it, clicking on a scrolling arrow (or the scrollbar itself) would cause the list to disappear. Why? Because the input field’s onblur() would fire and close the list.

So I looked through the code a bit, and came up with this simple solution.

First: make sure that you save a reference to the Ajax.Autocompleter object that you create:

var oAutocompleter = new Ajax.Autocompleter()

Second: define an undocumented option in the options parm list:

onHide:	fOnHide

…where fOnHide is a function we’ll define later.

Third: add a boolean var to the Autocompleter object:

oAutocompleter.lcl_bActive = false;

Fourth: apply some event handlers to the div that is populated with the results:

oAutocompleter.update.onmouseover = function() {oAutocompleter.lcl_bActive = true;};
oAutocompleter.update.onmouseout = function() {oAutocompleter.lcl_bActive = false; oAutocompleter.element.focus();};

Fifth, and finally: define the fOnHide() function:

function fOnHide() {
  if (!oAutocompleter.lcl_bActive) {
    new Effect.Fade(oAutocompleter.update,{duration:0.15});
  }
}

Now – how does this work? (good to know, since I’m sure it’s inevitable that it won’t work for somebody or at some point in time)

It seems that somebody ends up calling the onHide() function of your instantiated Ajax.Autocompleter object whenever the input field’s onblur() fires. Normally this results in an Effect.Fade being instantiated, and your result list fades out of existence.

We interrupt that process – instead of calling the Effect.Fade() directly, we first check to see if a flag has been set.  The flag is set to true whenever the pointer is hovering over the list – manipulating the scroll buttons or scrollbar or just hanging out.  In those instances, the onHide will essentially do nothing, and your autocomplete list will not disappear.

Once you move away from the list, the flag is reset and, crucially, focus is returned to the input field.  This is done so that the field’s onblur() can fire when appropriate, as it would have done before we made any code changes.  Now, when the field loses focus, the fOnHide() function will perform the default task of running the Effect.Fade() function.

Easy breezy 🙂

The Ultimate smartphone?

No, I’m not talking about my month-old HTC Desire Z.

I’m thinking of the new (and as yet unreleased) Motorola Atrix 4G.  Besides the dual-core silicon running under the hood – something that seems destined for all new high-end smartphones btw – the big news for the Atrix is the “webtop” app and docking abilities. Said abilities allow your little pocket burner split personalities, shedding its smartphone skin for a laptop visage or home multimedia center duty.

Certainly sounds cool, and brings new weight to the notion of a converged device. But the problem, as I see it, is that you can’t deal with all of these persona at the same time.

Here’s the skinny.

Right now I’m in Starbucks, and I was thinking to myself; “Man, it would be nice if I had my laptop with me.” Then I thought to myself; “Hey, if I had a Moto Atrix I’d be set!”

But I wouldn’t.

For the Atrix to be useful in this situation, I’d need the laptop dock. And if I’m going to carry the dock around with me, then how is that any better or worse than carrying my netbook with me? Without the laptop dock the Atrix is no more useable than my Z.

Wait, it gets worse.

With the laptop dock, and running the webtop app, I can’t do cool things like run a Java-based SSL tunneling client to connect to my home network. And that’s just one example. What about running a development IDE?

Sure, you can run Google Docs and do any number of other cool web-based thingamijigs. But a real laptop the Atrix is not.  It’s somewhere between a netbook and a tablet, in my humble estimation.

Then there’s the multimedia dock. And again, you’ve got to look at it for what it really is. Today’s tech push into your living room is all about net-connectedness. Watch Netflix on your TV. Watch Hulu. Do it fullscreen, in full HD, at 24 to 30 FPS.

Now, you may be able to get close to some of that using the Atrix’s web-browser in webtop mode, but the jury’s still out on fullscreen performance. And that’s crucial.

You can get the fullscreen, full HD, full FPS experience using the media player in webtop mode, but my research leads me to believe that that’s for local content only. No Hulu, no Netflix, no vids from your Windows Home Server or other DLNA-graced devices.

So again – what’s the point?

Don’t get me wrong; the Atrix is a great example of pushing the boundaries when it comes to smartphone design and use. But I fear that the implementation, while abounding in gee-whiziness, doesn’t really give us anything that we can’t already get (better) by carrying the same amount of hardware. Factor in efforts like Firefox’s Weave or Opera’s Link, and the lines between your desktop and your mobile are already being blurred (I’m thinking along the lines of the vaunted Connected Client effort).

I have no doubt that people will drop large dollar bills to get an Atrix and its assorted docks. But I truly wonder if their digital lifestyles will be improved as a result of it.

Apps that make my life easier (Droooooid Part 3)

I’m still in the process of loading up my Desire Z with cool and useful apps.  Here’s the list of ones that have made the cut, and are worth a mention.

XiiaLive

Trés useful for streaming audio from home.  You can’t change the streaming settings in the “Lite” version, but even with an 8-second buffer it’s entirely tolerable.  And the audio quality over Bluetooth A2DP is very good, much better than it was on the Touch Pro (and this is without having to tweak anything – and that’s a Very Good Thing(tm))

k9mail

Built off the source code of the native Android Mail application, this app adds such useful features as being able to specify your Sent and Trash folders when using IMAP.  This is a boon for me, since I use IMAP extensively for my personal mail.  This program finally plays nice with my mail server after some initial struggles.

KeepScreen

I couldn’t understand why my Z would turn off when Google Maps was running.  The Touch Pro never did this.  It makes sense that if you’ve got Maps open, chances are you’re navigating somewhere and you want Maps to stay open.  [Personal note: see if Google Navigation keeps the screen on]  Keep Screen will keep the screen on when certain applications are running in the foreground.  Works a treat.

PaderSyncDAV

This program allows me to access the files on my home network.  I recently uncovered a bug where I seem unable to upload files from this app.  I’ve contacted the dev and I’m hoping for a (speedy) resolution. [Update Jan 21, 2011: Worked with the dev to get this problem fixed, so it’s all good!]

Tasker

Back in the WinMo days I used Mortscript and some custom C programs to perform event-driven tasks.  In the Android world, I’ve found that Tasker can do many things I did with Mortscript – and more.  So far I’m using it to unlock the orientation sensor when I’m in the car, and to start Google Navigation when I’m in the car and external power is connected.  These two tasks alone are worth the US$6.

Pure Calendar/GridCalendar/Messenger Widget

This trio of widgets fulfills my desire to see consolidated info at a glance on my homescreen(s).  Pure Messenger Widget, in particular, is quite useful as a means to get my incoming Twitter, Facebook, GMail and SMS updates all in one place, as a scrollable list no less.  Think of FriendStream for HTC Sense, only more customizable.

Droooooooid – Part Deux

Many, many, many hours of work has been done since this post was posted.

The results are mostly good.  As it stands now, I’ve basically got all of the core functionality I had on the Touch Pro – plus more, obviously.  And a nice consequence of this work is that much of the processes I’ve used in the past are now somewhat simplified.

In particular, I’m quite stoked about the uber-quick WebDAV access and client-less access to the intranet.  That last one needs some security work, but it’s there and it’s a beaut.  I’m yet to test streaming from the Home Audio streaming zones, but that’ll likely happen in 1 hour or less (once I go out for lunch).

I also made some useability improvements to the fileserv, which is what allows me to send links and files to the phone.  Google has a Firefox and Chrome extension that allows you to do this, but I prefer to roll my own solutions.  Additionally, this one doesn’t require any special software that somebody isn’t likely to already have.

Truthfully the fileserv stuff has had a “todo – enhanced security” line item for some time now.  That’s not resolved yet, but I imagine I’ll be using this facility more going forward and that alone may prompt me to get all this security stuff nailed down.

So what’s not working?

Mail isn’t 100%.  I worked with the developer of my server software to iron out one bug, and that was great, but as it stands now the native Android Mail client is missing some core functionality (ability to set trash/sent folders is the biggest, and the ability to filter IMAP folders) so I’m using the k9mail client instead.  Unforunately that client isn’t doing two-phase deletions/moves properly.  So for now I’ve got to use both clients.  A bummer for now, but hopefully it will get resolved soon.

What’s changed?

My particular brand of push notifications have changed.  You may or may not know that certain events (types of mail, surveillance) generate notifications on the phone.  In the WinMo world these notifications were delivered via SMS. Some would automagically (courtesy of PushEffect) tell Outlook Mobile to connect and sync my mail.  That last little bit simulated push email, and while I have no such facility now (but may again play with IMAP IDLE) it’s sort of a moot point now; both Mail and k9mail are so fast to sync folders that I now do the process manually, which is really no more effort than opening Outlook Mobile to read the mail I just got notified about.  I could probably spend some time in the future to replicate the sync functionality on Android, to operate in the same manner it did in WinMo.  Chances are that that won’t happen any time soon though.

So anyway, the notifications were delivered by SMS, by way of an Email-to-Text gateway supplied by my mobile carrier.  And they being the gougers they are, they decided that I had to pay for the privilege of getting these missives directly instead of having their own servers cache them and send me stupid “you have mail, reply R to read” SMS messages in their stead.  That always bothered me, and while I took some baby steps in the past month or so to get around that in Windows Mobile, I now utilize a GMail account in Android to get around it completely.

Both methods offer quick delivery of said notifications.  The GMail method has no length restrictions, which is good, while the SMS method had no data usage, which was good.  But with 6GB of data allotment each month, data usage isn’t really a concern.  What is a concern is roaming data.  With data-based notifications I simply won’t get any notifications at all if data is off due to roaming.  So I’ve got to think about this one.

Nevertheless, being able to get of the monthly charge is a  Good Thing(tm).  Worst case?  I switch back to SMS-based notifications while out of the country, and use a little hack to deliver the essence of the message without the direct-email-to-text feature.

So that’s that for now.  I feel that a Part 3 will be coming soon.  Stay tuned.