Runkeeper and my Garmin GPS

The problem

I am an avid runkeeper.com user. I log every hike, every bike ride, everything. I have used two devices: my android phone, and a Garmin Vista hcx GPS. The later is particularly useful for my long hikes: It is sturdy, waterproof and batteries last very long.

My problem with my Garmin GPS is that something in the way it logs my tracks makes Runkeeper assume I am doing constant pauses, and resuming a few meters later. This takes all calculations (distance, total climb) very far off what I get from what I get from Mapsource or Google Earth. And this cannot happen in Runkeeper where I expect to have every possible statistic of my activities!

This has been happening on my activities for the last 2 or 3 years of Runkeeper usage...

Why now?

Long ago (Aug 2011) I reported this issue to the support team. This was their reply:
From: Jake
Subject: Problem in climb calculation

Hernan, no known issues with Climb calculation, it appears you have a large number of pause/resume points in this activity, which may be what accounts for the discrepancy.  It looks like there were gaps between pause points that may have accounted for large elevation change.  If you track an activity continuously, do you notice the same discrepancy?

Here'e how we calculate climb....http://support.runkeeper.com/kb/troubleshooting/how-does-runkeeper-calculate-elevation-and-climb

View this Discussion online: http://support.runkeeper.com/discussions/problems/10363-problem-in-climb-calculation

Unfortunately the link is now broken -- they have moved their forums, and apparently, garbage-collected old tickets. A total shame :(

This is why I have been using my phone more and more, and my GPS unit less. But after a 3-day hike where I really had to use the GPS, I decided to fix the issue...

The hack

The idea is really simple: Just make the tracks I upload "smoother", by adding points whenever two points differ in time (or distance, or elevation?) beyond what Runkeeper would consider a 'break'.

Normally, I load all my tracks to a .gdb file (from MapSource), and run scripts to export each track to all .gpx, .tcx and .kml -- just in case! Then I upload the .gpx or .tcx to Runkeeper. So I decided to do some post-processing of these files.

I found gpxpy, a Python library to read and write GPX format. One of the examples in the github page already parses out a file, and even prints some statistics -- almost exactly what I need. So what was left to me to do was to complete this, and I came up with the python script I am attaching at the end.

Results

I have uploaded and re-uploaded my 3-day hike, and now the numbers look much more accurate. There are still some pauses, but they may be totally true -- these are 10-hour hikes, and they do have pauses -- just not every minute.

Here is an image of my activity before fixing the track:

and this is the same activity, after the fix:

note the improvements: 1608m climbed (against 439), 12.95km instead of 11.07km, and a much cleaner curve!

I would love to receive feedback from other users -- as well as Runkeeper staff! For me, it is not a big deal if my GPS is too old or incapable or producing the correct input formats for them. I will keep loving and using Runkeeper!

If you found this article useful, or need help trying to use the code yourself, please leave a comment!

PS: The source code

Bits from the past

A few weeks ago my father found in a backup disk some old texts he had written years ago: composed in WordStar for DOS; likely in our first computer, a PC XT (8-12Mhz!). Digging these files we concluded they were from about 1991 or 1992.... Too long ago! All I can say in my defense is that... I was younger? A child? The fact is that I used that WordStar version a lot.

Unfortunately these files were unreadable by any word processor I use and tried. Import filters promised a lot, a none of them worked for me. So it deserved some deeper digging.... First of all, for nostalgy sake, this is how WS looked like (now within a "modern" Win XP, running virtualized in my linux):

Now back to importing these files: I found the site wordstar.org with plenty of information, but most downloads were for Windows, and also most were not free. Here is a list of downloads. I tried some of them (under Wine), like: WS-Con, WSRTF and a few more. None of them were fully working; most have problems with accents or whatever encoding these files used.

Fortunately I found a text from in site, which describes the file format. This format is quite simple, and has a nice design allowing to extract "most" of the text by just looking the lowest 7 bits of each bytes, and discarding everything with the 8th bit set. If you want formatting, you would have to interpret those high bytes, though they are not too complex and we used very few in our old texts.

So I read this and wrote a python script to process them... The first few attempts were mangling, again, all my accents and the 'ñ' character (these are in Spanish) so I had to start digging at ascii codes. I have almost tatooed in my memory that 'ñ' = 164, after typing "Alt-1-6-4" so many times in DOS. (There were only US keyboards by that time... and I still use them). But a character 164 means something else in python or in nowadays encodings... sometimes as bad as:

>>> chr(164)
'\xa4'
>>> chr(164).encode('utf-8')
Traceback (most recent call last):
  File "", line 1, in
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa4 in position 0: ordinal not in range(128)

while the 'ñ' has different codes today:

>>> 'ñ'
'\xc3\xb1'   ### This is UTF-16 I think
>>> 'ñ'.decode('utf-8')
u'\xf1'      ### Its UTF-8 encoding

so which was the correct encoding? There is a small note in this page, saying "how to type in microsoft windows", and later a note saying DOS was using "codepage 437". That's cool, I had already found the list of all encodings python was bundled with in /usr/lib/python2.6/encodings (and there is indeed a file cp437.py)

So the real key was to do something like chr(164).decode('cp437'). That returns the unicode string u'\xf1', which is the "real" 'ñ'. That made the trick, and the script was done.

As a side note: I found some more characters I could not filter out initially: Double-byte codes like ESC-'4' or ESC-'5' around words... what was that? Some sleepy neuron remembered that we use to have a Star NX-1001 (multifont!), and I suspected that was a printer code. In fact, the manual (which still exists! go Star!) says those are the start of italicized text, and the return to normal font face. So that's not part of WordStar format, it's another problem - the way we handled formatting in our printer. (It was good to remember that, our best printer, again!).

Now if you read this far you must be a nostalgic looking for old memories, of really looking to translate a wordstar file. I uploaded the script to http://pastebin.com/pfY8Dbgv - it converts to plain text or a basic HTML. I hope it helps you!

Google Codejam 2010 participants feeds

On May 7th the Google CodeJam 2010 started, with about 15,000 contestants in the Qualification round! If you are competing, you surely hit this blog looking for some stats.

I have not found any feed (yet) about participants. All I found was this thread from the google group: where Igor N. from Google said: we still have the AJAX requests that the scoreboard page uses to grab 20 rows at a time.

So I went ahead and grabbed, 20 rows at a time.... One bash command line that iterates and gets 15000/20 small json files, and a few python lines to parse and pretty print them. Here is how my output currently looks like (the scores for each problem overflow to the right):

## Country: Argentina
#210 (99): ged       [ 4:42:08]  [s 2:16:12 L 2:16:57]  [s 4:37:31 L 4:38:08]  [s 3:24:24 L 3:25:22]
#326 (99): lrearte   [ 6:47:22]  [s 6:46:17 L 6:47:22]  [s 6:43:08 L 6:44:32]  [s 6:34:26 L 6:36:44]
#408 (99): aurinegro [ 8:29:45]  [s 0:43:37 L 0:44:24]  [s 1:09:52 L 8:25:45]  [s 2:12:08 L 2:13:43]
#454 (99): axelbrz   [ 9:16:12]  [s 4:35:26 L 4:41:16]  [s 9:04:06 L 9:08:12]  [s 2:54:08 L 2:58:50]
#466 (99): dodi      [ 9:28:48]  [s 5:34:01 L 5:37:01]  [s 9:20:20 L 9:28:48]  [s 7:27:13 L 7:31:59]
#709 (99): fidels    [12:40:11]  [s 0:12:58 L 0:15:09]  [s 3:35:34 L 3:39:19]  [s12:37:54 L12:40:11]
(....)

There are more fields to show; basically everything shown in the scoreboard is obtained from these feeds so you can look for the extra data in the json files.

Feel free to use your script for yourself. I would love to hear similar pages or your own usage of these feeds.


Here is the python script (note that the bash command line to get the results is in a comment)

#!/usr/bin/python
import os
import json

## Run first:
## for i in `seq 1 20 10500`; do wget -O $i.json "http://code.google.com/codejam/contest/scoreboard/do?cmd=GetScoreboard&contest_id=433101&show_type=all&start_pos=$i&views_time=1&views_file=0&csrfmiddlewaretoken="; done
def time(pty):
  return "--:--:--" if pty<0 else "%2d:%02d:%02d" % (pty//60//60, (pty//60)%60, pty%60)
def problems(arr):
  s = ""
  c = 's'
  for t in arr:
    s += '   [' if c=='s' else ' '
    s += c+time(t)
    if c=='L': s += ']'
    c = 'L' if c=='s' else 's'
  return s

country='Argentina'
print "## Country: "+country

i=1
while True:
  try:
    f = open(str(i)+".json")
  except:
    exit
  if (f.closed): 
    break
  s = f.read();
  f.close()

  obj = json.loads(s)
  for coder in obj['rows']:
    if coder['c']==country:
      print "#%5d (%3d): %-30s  [%s]    %s" % (coder['r'], coder['pts'], coder['n'], time(coder['pen']), problems(coder['ss']))

  i += 20

Installed Python module (thrift) not being picked up

I have been using thrift for some time (compiled from source code), and a recent upgrade of my linux to Jaunty broke something --- I think.

My symptom was trying to run a script that imported Thrift, receiving this message:

Traceback (most recent call last):
 File "./Cassandra-remote", line 11, in <module>
   from thrift.transport import TTransport
ImportError: No module named thrift.transport

However that module file exists in my disk: /usr/lib/python2.6/site-packages/thrift/transport/TTransport.py

It took a while to realize that this path was not being inspected by python:

$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
'/usr/lib/python2.6',
'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/usr/lib/python2.6/dist-packages',
'/usr/lib/python2.6/dist-packages/Numeric',
'/usr/lib/python2.6/dist-packages/PIL',
'/var/lib/python-support/python2.6',
'/var/lib/python-support/python2.6/gtk-2.0',
'/usr/local/lib/python2.6/dist-packages']

(note that "site-packages" is not in the list). I assume that I configured my thrift code when the paths were different in my system, and some upgrade changed python directories. So when I "make install" thrift, they are still copied into site-packages. Now, having thrift already installed there, it is a matter of adding this dir to the search path. This is done with sys.path.append, according to python's documentation:

>>> sys.path.append('/usr/lib/python2.6/site-packages')
>>> sys.path
['',
'/usr/lib/python2.6',
(...)
'/usr/local/lib/python2.6/dist-packages',
'/usr/lib/python2.6/site-packages']
>>> from thrift.transport import TTransport
(No error message now)

The "sys.path.append" worked, by I need to persist this change. This can be done by changing the variable PYTHONPATH, I am adding this line to my .bashrc:

export PYTHONPATH=/usr/lib/python2.6/site-packages

And that's it. Any new console (on my user at least) gets this path and I can import the thrift module now.

I wonder how to make this change available to all users -- where is the "default" PYTHONPATH defined?