Marginal Structure |

Getting Started – The Raw Data

A lot of sites talk about working with KML files or GPX files, but there arent many that talk about how you get your raw data from the GPS logger into a format you can work with, or what the raw data means. i think this is partially a result of the fact that many GPS loggers use proprietary formats and custom software for this purpose. I think it’s probably also true that not very many people are interested in the raw data, since they can use existing applications to do the things they want, like tag their photos. They might never look at the raw data at all.

But if you’re interested in doing any sort of rigorous manipulation or analysis of this data, you want to know what you’re getting and what it means. And preferably, you want to do this before it’s converted to another format. The conversion could be a lossy process (it might not preserve all the data), you might want to customize the output or write your own parsers, etc.

Several GPS data loggers have the option of writing data in the text-based NMEA format. This seems to be the lowest-level text based format– it is not a markup or presentation format, but contains the raw so-called ’sentences’ of GPS data.

NMEA Format

The NMEA format records different types of data in what it calls ’sentences’. Each sentence represents a different type of information. Every time your GPS device take communicates with a satellite, several different sentences can be recorded.

Think of this like packets on a network. There are many different kind of packets which flow across a network. Each packet type is meant to encompass different information, and different applications or devices can exchange different types of packets. Most, of course, use several kinds of packets to accomplish their full goal.

Such is the case with your GPS device, too. Depending on what it’s designed for, it will be configured to record different NMEA sentences. Here is a quick reference with examples, of some of the main NMEA sentences. Most information here is from this very thorough and excellent site on the NMEA format:

$GPRMC:  this is the basic, minimum ‘recommended’ data to collect.

• $GPRMC,212552.196,A,3718.0148,N,12203.3034,W,0.18,86.11,251008,,,A*4E
  • 212552.196 is UTC time of position reading
  • A=valid, V=warning
  • 3718.0148,N,12203.3034,W are the lat and long
  • 0.18 is the ground speed, in knots (!) (it is a maritime standard)
  • 86.11 is the ‘true course’ heading
  • 251008 is a datestamp (25/10/08, or october 25 2008)
  • the two blank fields are the magnetic variation, and direction (E/W)
  • the last field, A is a mode indicator (A=Autonomous, D=Differential, E=Estimated, N=Data not valid). my understanding is that this field is only present in NMEA version 3.
    
  • the final field (*4E) is a checksum

$GPGGA: information on the GPS ‘fix’ used in obtaining the positioning data.

• $GPGGA,212553.196,3718.0147,N,12203.3030,W,1,03,2.1,25.9,M,-25.9,M,,0000*52
  • 212553.196 is the time in UTC
  • 3718.0147,N,12203.3030,W is the lat and long at the time of the fix
  • 1 is a flag meaning ‘GPS Fix’. 0 means invalid, and 2 means DGPS (not sure what that is)
  • 03 is the number of satellites in view (ie involved in determining the GPS data)
  • 2.1 is the relative horizontal position accuracy
  • 25.9 is the altitude in meters above mean sea level.
  • the remaining fields are ‘Height of geoid above WGS84 ellipsoid’ (??), time since last DGPS update (usually blank), DGPS last station ID,
  • *52 is a checksum for the sentence as a whole.

$GPGSA: a sentence recording the ‘DOP’ (dilution of precision) and active satellites involved in the reading.

• $GPGSA,A,2,04,02,12,,,,,,,,,,2.4,2.1,1.0*32
  • A is hte mode (A=automatic or M = manual)
  • 2 is the type of fix (1=fix not available, 2=2d, 3=3d)
  • positions 3-14 are effectively IDs of the satellites involved in the fix. the number of entries here should correspond with the ‘number of satellites in view’ value from the $GPGGA sentence, above. note that here there are 3. the IDs are actually called PRNs or ‘pseudo random numbers’. the active GPS satellites each have PRNs between 1 and 32.
  • the next three numbers (2.4, 2.1, 1.0) are the positional DOP, horizontal DOP, and vertical DOP of the positional information recorded. my understanding is that the DOP is like a quality measurement. the closer the satellites used in the coordinate reading are, the poorer the quality of the data and the higher the ‘dilusion’ is. according to the wikipedia article on DOP, values in the range 1-3 are ‘excellent’, 4-6 are ‘good’, etc.
  • as usual the last value *32 is a checksum.

$GPVTG:

• $GPVTG,72.05,T,,M,0.11,N,0.2,K,N*30
  • 72.05 T is the true course ‘made good’
  • the next field (which is blank here) plus the ‘M’ represent the ‘magnetic course made good over the ground’, in degrees.
  • 0.11N is ground speed in knots
  • 0.2 K is ground speed in km/h
  • the final field is another mode indicator, same as the one at the end of $GPRMC sentence:(A=Autonomous, D=Differential, E=Estimated, N=Data not valid). again, my understanding is that this field is only present in NMEA version 3.

Devices

If you are looking in particular for a life logging device, something that will always be on and with you, will be generating large amounts of data, and on 24 hours a day, then rechargeable Li-Ion (or similar) battery and expandable memory capacity (miniSD or something) are both key. The AMOD AGL 3080 was very well reviewed and performs great, but even at 128MB, it only stores about 72 hours worth of data (on the one measurement/second setting) and it’s 3 AAA batteries last less than 24 hours if you are using rechargeables. You basically have to carry around a battery charger everywhere you go, and always have multiple sets of batteries on hand. Sub-optimal for life logging. For what it’s worth, and this may just have been me, but the file system got corrupted after about 6 weeks worth of use and became unusable. (Because I got a new device for the other reasons listed, I didnt bother to try reformatting the filesystem).

For me, a non-proprietary interface is also paramount. Something that mounts as a USB. If you use a device with a memory slot, then it become less of an issue since you can copy the data directly off the miniSD card. but that brings up the next requirement, which is something that records to a text-based format (as mentioned above, NMEA is the main format i have seen along these lines). if you ask me, you should never rely on a proprietary interface to do science! not just philosophically, but you also risk getting in a position where the driver or software you need isnt supported anymore.

Logging frequency and modes. The option of logging every 1 or at a minimum every 5 seconds is important, but most decent devices seem to do this now. Which nmea sentences does it support? think about what you want to do with the data, and make sure you’ll have what you need. Position-related sentences and sentences that report on the quality of your measurements are both going to important for knowing the accuracy of your data.

Be careful of devices that say they ’support format X’– this may only mean their software will convert to it. You want to make sure it will natively write to the format you want.

I have seen comments that binary formats such as SBP on the genie BGT-31 are smaller and more accurate. i think the size difference is worth the flat file format, but i dont know where the accuracy differences would come from, except perhaps rounding errors in conversion? Or how pronounced the accuracy differences are. If you have experience with this, please let me know. (One day, I’ll compare coordinate data from binary and text formats and post here…)

Tools and Frameworks

• gpsbabel (”GPSBabel converts waypoints, tracks, and routes between popular GPS receivers and mapping programs. It also has powerful manipulation tools for such data.”)
• Geospatial Data Abstraction Library (GDAL). from matt: ”GDAL is the swiss army knife of geospatial data processing. pretty much everybody uses it, and it does everything pretty well … it’s like one of those gag swiss army knives that’s like six inches thick and has way too many tools to wrap your brain around.”
• Geodjango
• SpatiaLite

Coming Soon:

• Operational Considerations (2Mb or 2Gb file sizes? Where to store? How to backup?)
• Converting to KML and Viewing your paths in Google Earth
• Storing your data in a Database
• Writing Scripts to Analyze your data.