<?xml version="1.0" encoding="iso-8859-1"?>
<rss version="2.0"
 xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
 xmlns:dc="http://purl.org/dc/elements/1.1/"
 xmlns:admin="http://webns.net/mvcb/"
>
<channel>
<title>Kurt's Weblog</title>
<link>http://schwehr.org/blog/rss.xml</link>
<description>electronic work log</description>
<dc:language>en-us</dc:language>
<dc:creator>Kurt</dc:creator>
<dc:date>2013-06-16T13:27:17-04:00</dc:date>
<admin:generatorAgent rdf:resource="http://nanoblogger.sourceforge.net" />
<item>
<link>http://schwehr.org/blog/archives/2013-06.html#e2013-06-16T12_57_01.txt</link>
<title>autoconf/automake changes to mb-system for conditionally removing GSF</title>
<dc:date>2013-06-16T12:57:01-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[This is my first really deep dive into autoconf / automake.  I have
been a heavy user of configure scripts and could sort of hack my way
through minor changes, but I feel like I can actually do some useful
work on autoconf and automake scripts now.  I just posted a very large
patch on the mb-system mailing list that lets people use --without-gsf
to build mb-system without the Generic Sensor Format library by SAIC,
code which has no license.  Here is a version of that email.
<br /><br />
<a href="https://gist.github.com/schwehr/5792674">https://gist.github.com/schwehr/5792674</a>
<br /><br />
This is a first step towards removing GSF to allow MB-system to be
packaged by Hamish for submission to RedHat linux and Debian linux.  I
think this should make copyright sanitization job of the packaging
just require deleting the gsf subdir, making a new tar, and then
including --without-gsf in the build process.
<br /><br />
I've got a patch that I believe successfully builds with or without gsf.  Would really appreciate very detailed review of this patch.  It builds for me on Mac OSX 10.8 and the present of libgsf in the shared libraries list looks okay, but I haven't tried running any files through yet.  Watch through of the patch below.  I'm super detailed below to hopefully help in the review and because this is the first time I've done something like this with autoconf/automake and I'm double checking what I did.
<br /><br />
The patch is kind of large because it contains deleting mb_config.h which shows as a "-" changes for the entire file.
<br /><br />
<b>Notes from Hamish about GSF issues of being unlicensed and submission to deb & rhel/fedora</b>
<br /><br />
Kurt, the optional-system's proj4 and GFS switches will certainly not
get past the debian ftp masters, actually the deb packaging will need
to rip out any license problematic stuff and make a new tarball, since
all original code packages are hosted on the debian servers, and they
as a legal entity need to be able to guarantee that they've got full
copyright clearance for anything they redistribute.
<br /><br />
<b>TODO - header.h.in conversion</b>
<br /><br />
I did not convert any of the C headers that have to change to their respective .in template version.  e.g.
<br /><br />
mb_format.h -&gt; mb_format.h.in
<br /><br />
That means that building against installed headers & libs is not going to work right.  e.g. 
<br /><br />
WITHOUT gsf means that prototype for mbr_register_gsfgenmb should not be in mb_format.h at all (or it should at least be commented out)
<br /><br />
<b>TODO - should we delete, rename or leave alone structure members for gsf?</b>
<br /><br />
In order to catch the use of gsfid, I renamed this struct member to _gsfid_placeholder when GSF not defined:
<pre>#ifdef WITH_GSF
        int     gsfid;          /* GSF datastream ID */
#else
        int     _gsfid_placeholder;
#endif</pre>
Would it be better to leave it alone or could we just drop it like this, which would change the size of the structure (possibly causing problems with things like what ???):
<pre>#ifdef WITH_GSF
        int     gsfid;          /* GSF datastream ID */
#endif</pre>
<b>mb_config.h</b>
<br /><br />
This file should not be checked into svn.  It doesn't cause trouble if
you build on top of the source tree, but if you do VPATH style
building, it gets included before the new mb_config.h that gets
written into the build tree.
<br /><br />
<b>How I was building</b>
<pre>svn co svn://svn.ilab.ldeo.columbia.edu/repo/mb-system/trunk mb-system
cp -rp mb-system{,.orig}
<!-- -->
mkdir build
cd build</pre>
WITHOUT gsf
<pre>(cd ../mb-system; ./autogen.sh); rm -rf src Makefile config.{status,log}; \
../mb-system/configure --with-netcdf-include=/sw/include --with-gmt-lib=/sw/lib \
--with-gmt-include=/sw/include --disable-static --without-gsf && make -j 2
<!-- -->
(cd ../mb-system; ./autogen.sh); rm -rf src Makefile config.{status,log}; \
../mb-system/configure --with-netcdf-include=/sw/include --with-gmt-lib=/sw/lib \
--with-gmt-include=/sw/include --disable-static --with-gsf=no && make -j 2</pre>
WITH gsf (no option, means include/build with gsf)
<pre>(cd ../mb-system; ./autogen.sh); rm -rf src Makefile config.{status,log}; \
../mb-system/configure --with-netcdf-include=/sw/include --with-gmt-lib=/sw/lib \
--with-gmt-include=/sw/include --disable-static && make -j 2
<!-- -->
(cd ../mb-system; ./autogen.sh); rm -rf src Makefile config.{status,log}; \
../mb-system/configure --with-netcdf-include=/sw/include --with-gmt-lib=/sw/lib \
--with-gmt-include=/sw/include --disable-static --with-gsf && make -j 2
<!-- -->
(cd ../mb-system; ./autogen.sh); rm -rf src Makefile config.{status,log}; \
../mb-system/configure --with-netcdf-include=/sw/include --with-gmt-lib=/sw/lib \
--with-gmt-include=/sw/include --disable-static --with-gsf=yes && make -j 2</pre>
Check for libgsf shared library being linked in as a double check
<pre># WITHOUT gsf
otool -L src/utilities/.libs/mbinfo  | grep gsf
<!-- -->
# WITH
otool -L src/utilities/.libs/mbinfo  | grep gsf
#	/usr/local/lib/libmbgsf.0.dylib (compatibility version 1.0.0, current version 1.0.0)
</pre>
<b>Building the patch</b>
<pre>cd ../mb-system
<!-- -->
# revert files with don't need to patch
find . -name Makefile.in | xargs rm
rm -rf configure autom4te.cache aclocal.m4 
svn up
<!-- -->
# this file needs to leave svn!
rm src/mbio/mb_config.h
<!-- -->
cd ..
diff -ruN --exclude=.svn mb-system.orig mb-system > mb-with-gsf.patch</pre>
<b>Changes to how configure.in and the resulting configure</b>
<br /><br />
The options now look like this:
<pre>  --with-otps_dir=DIR        Location of OSU Tidal Prediction Software
  --without-gsf           Disable unlicensed SAIC Generic Sensor Format (GSF)
  --with-netcdf-lib=DIR	Location of NetCDF library</pre>
The configure.in changes:
<pre>AC_MSG_CHECKING([whether to build with Generic Sensor Format (GSF)])
AC_ARG_WITH([gsf], AS_HELP_STRING([--without-gsf], [Disable unlicensed SAIC Generic Sensor Format (GSF)]), [ ], [with_gsf=yes])
AC_MSG_RESULT([$with_gsf])
<!-- -->
AS_IF([test x"$with_gsf" = xyes], [AC_DEFINE(WITH_GSF, 1, [Build with GSF])])
AM_CONDITIONAL([BUILD_GSF], [test x"$with_gsf" = xyes])</pre>
The above is different than all the other options.  I tried to use the provided autoconf and automake macros as much as possible.  The above does 3 things:
<ul>
<li>convert the "#undef WITH_GSF" in mb_config.h.in to "#define WITH_GSF 1" or nothing.</li>
<li>Set the value of BUILD_GSF so that the Makefile.am files are properly converted to Makefiles with control of the "if BUILD_GSF" tweaks</li>
<li>The combination of using all the AS_ macros means that --quiet can work as intended. </li>
<li>Anything other than no option, --with-gsf, or --with-gsf=yes is treated as a leave out GSF</li>
</ul>
TODO: should we rename BUILD_GSF to WITH_GSF so that they match?
<br /><br />
The final line of configure.in shows how autoconf wants us to print summaries:
<pre>AS_ECHO(["Build with Generic Sensor Format (GSF) Support: $with_gsf"])</pre>
is a lot simpler than:
<pre>if test x"$with_gsf" = xyes ; then
	echo "Generic Sensor Format (GSF) Support: Enabled"
else
	echo "Generic Sensor Format (GSF) Support: Disabled"
fi</pre>
<b>Do not build the gsf subdirectory</b>
<br /><br />
In src/Makefile.am, I conditionally insert the gsf directory into SUBDIRS.  I normally use "SUBDIRS +=", but I wanted to follow the convention already used in the file.
<pre>if BUILD_GSF
  XBUILD_SUB_GSF = gsf
endif
<!-- -->
SUBDIRS = $(XBUILD_SUB_GSF) surf mr1pr proj mbio mbaux utilities gmt otps macros $(XBUILD_SUB) $(XBUILD_SUB_GL) man html ps share</pre>
<b>conditionally add gsf files</b>
<br /><br />
I put gsf only source files in with the "+=" syntax of Makefile.am.  e.g. in mb-system/src/mbio/Makefile.am:
<pre>include_HEADERS = mb_config.h \
        mb_format.h mb_status.h \
        mb_io.h mb_swap.h \
<snip>
        mbf_mbarirov.h mbf_mbarrov2.h \
        mbf_mbpronav.h mbf_xtfr8101.h
<!-- -->
if BUILD_GSF
  include_HEADERS += mbsys_gsf.h mbf_gsfgenmb.h
endif</pre>
<b>use mb_config.h to conditionally compile code</b>
<br /><br />
For example in mb_close:
<pre>#include "gsf.h"</pre>
Becomes:
<pre>#ifdef WITH_GSF
#  include "gsf.h"
#endif</pre>
<b>The include guard situation in mb_system is a little inconsistent</b>
<br /><br />
The normal style of include guards is that the actual head should do the include guard, not the file that does the #include.
<pre>-#ifndef MB_DEFINE_DEF
+#include "mb_config.h"
 #include "mb_define.h"
-#endif
-
-#ifndef MB_STATUS_DEF
 #include "mb_status.h"
-#endif</pre>
<b>As discussed above, I did this in mb_io.h to help make sure I didn't miss anything</b>
<pre>+#ifdef WITH_GSF
 	int	gsfid;		/* GSF datastream ID */
+#else
+        int     _gsfid_placeholder;
+#endif</pre>
I then #ifdefined WITH_GSF around any gsf related code
<br /><br />
This is where I could really use others taking a close look.  e.g.
<pre>+#ifdef WITH_GSF
 	mb_io_ptr->gsfid = 0;
+#else
+        /* TODO: possibly set to -666 */
+#endif</pre>
and
<pre>@@ -3072,6 +3086,7 @@
 	return(status);
 }
 /*--------------------------------------------------------------------*/
+#ifdef WITH_GSF
 int mbcopy_reson8k_to_gsf(int verbose,
 		void *imbio_ptr,
 		void *ombio_ptr,
@@ -3479,4 +3494,5 @@
 	/* return status */
 	return(status);
 }
+#endif
 /*--------------------------------------------------------------------*/</pre>]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-06.html#e2013-06-13T11_14_50.txt</link>
<title>How much of the world is deeper than X</title>
<dc:date>2013-06-13T11:14:50-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[I was asked for fraction of the ocean deeper than 20 fathoms.  That
gets tricky.  What would the differnce be if this were at MLLW vrs
MHW?  I welcome input from anyone specializing in geodesy, vertical
datums and tides.
<br /><br />
I prefer to work in units of chains.  So how much is deeper than 1.8
chains (aka -36.576m)... 97.1% of the WORLD (oceans + other things that might be
deeper but not in the ocean).  And this is relative to whatever
vertical datum topo30 uses.
<br /><br />
Here is using the topo30 2012 edition from David Sandwell and JJ
Becker.  First a warning.  There is a non-commercial license on the
topo30 data.  This is more restrictive than the GPL license that many
are familiar with.
<pre>Permission to copy, modify and distribute any part of this gridded
bathymetry at 30 second resolution for educational, research and
non-profit purposes, without fee, and without a written agreement is
hereby granted ...
<!-- -->
... use for commercial purposes should contact the Technology Transfer
& Intellectual Property Services, University of California, San Diego ...</pre>
<pre><b>wget ftp://topex.ucsd.edu/pub/srtm30_plus/topo30/topo30.grd</b>
<b>gdal_translate topo30.grd topo30.tif</b>
<b>grdinfo topo30.grd</b>
<!-- -->
topo30.grd: Title: topo30.grd
topo30.grd: Command: xyz2grd -V -Rg -I30c topo30 -Gtopo30.grd=ns -ZTLhw -F
topo30.grd: Remark: 
topo30.grd: Pixel node registration used
topo30.grd: Grid file format: ns (# 16) GMT netCDF format (short)  (COARDS-compliant)           
topo30.grd: x_min: 0 x_max: 360 x_inc: 0.00833333333333 name: longitude [degrees_east] nx: 43200
topo30.grd: y_min: -90 y_max: 90 y_inc: 0.00833333333333 name: latitude [degrees_north] ny: 21600
topo30.grd: z_min: -11609 z_max: 8685 name: z
topo30.grd: scale_factor: 1 add_offset: 0
<!-- --> 
# z above is altitude in meters relative to ?
<!-- --> 
# install ipython, gdal-py and numpy
<b>ipython</b>
<!-- -->
import gdal
<!-- -->
src = gdal.Open('topo30.tif')
topo30 = src.GetRasterBand(1).ReadAsArray()
num_deep = (topo30 < -36.576).sum()  
'%.1f %% of the globe' % (num_deep / float(topo30.size) * 100)
<!-- -->
67.7 % of the globe
<!-- -->
num_ocean = (topo30 < 0).sum()
'%.1f %% of the ocean is deeper than 36.576m' % ( 100. * num_deep / float(num_ocean) )</pre>
<pre>'97.1 % of the ocean is deeper than 36.576m'</pre>
Minus issues with resolution, vertical datums, tides and other stuff,
97.1% of the world is deeper than 20 fathoms and 91.4% is deeper than
100 fathoms.
<br /><br />
<a
href="https://gist.github.com/schwehr/5789528">https://gist.github.com/schwehr/5789528</a>
- 
<a href="http://nbviewer.ipython.org/5789528">http://nbviewer.ipython.org/5789528</a>
<br /><br />
<img width="383" height="260" title="Topo30 histogram" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/topo30-hist.png"/>
<br /><br />
<img width="383" height="260" title="Topo 30 cdf" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/topo30-cdf.png"/>]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-06.html#e2013-06-11T21_21_54.txt</link>
<title>smart phones and similar devices in the field</title>
<dc:date>2013-06-11T21:21:54-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[It's great to see that work continues at CCOM on this topic:
<br /><br />
<pre>For people who think they can never have enough monitors! This is
the new panoramic display in the VisLab. The monitors are hinged so
they can be arranged as a flat wall or curved inward for a more
immersive experience. Among other things, the display will be used
with a ship simulator to study the potential benefits of hand-held
spatially aware navigation aids like smart phones. Thanks to Will
Fessenden for the nice photo!</pre>
<img width="600" height="247" title="CCOM large display" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/ccom-display-wall.png"/>
<br /><br />
Back in 1990, I got to tour Scott Fisher's virtual reality lab at NASA Ames.
After working on the Telepresence-ROV (TROV) vehicle in 1992, I was hooked on 
the idea of what might come when head mounted displays weren't so heavy that
they quickly made your neck hurt and didn't require a 5kW generator.
<br /><br />
Hiking around the Southern Snake Range doing geologic mapping, I
dreamed of devices that would sync up the mapping via hill top
repeaters (yeah, I had a HAM license).  Why waiting until the evening
to all copy our data to the master mylar sheet?  And if we could
capture field photos throughout the day, we could share best photos of
geologic units and particular features as we discovered them.  Instead,
I had to wait to get back to civilization to develop physical film.
<br /><br />
The next couple years, I had visions of laser range finders, stereo
cameras and mapping tablets as I worked in Yellowstone mapping hot
springs and in Mono Lake trying to get the TROV to do useful science.
Perhaps more scientists would take notes in the field if they were wearing
Google Glass and could just speak their observations.
<br /><br />
I did quite a bit at SIO in the 2002-2005 range with a 21 foot by 8
foot curved display and then when I got to CCOM in 2005, I proposed
hand held and head mounted augemented reality and heads up displays.
I got CCOM to pony up for the first smart phone bought on the NOAA
grant (an iPhone 3S for me).
<br /><br />
Roland and Tom have been doing some really neat development at CCOM and
it should be fun to see what comes.
<br /><br />
Some images from the first half of the 1990's that triggered some of
my thinking on wearable / portable devices and how they might help people
work in the real world.  If only GPS SA had been off back then!
<br /><br />
Sorry about the quality of the images.
<br /><br />
<img width="600" height="379" title="Mapping a hot spring in Yellowstone" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/1995-yellowstone-steep-cone.jpg"/>
<br /><br />
<img width="600" height="401" title="TROV in the news paper" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/trov-1995-tahoe-news-paper.jpg"/>
<br /><br />
<img width="600" height="285" title="house boat on mono lake" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/trov-house-boat-mono-lake-1995.jpg"/>
<br /><br />
<img width="296" height="450" title="TROV in Mono Lake" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/trov-in-mono-lake-1995.jpg"/>]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-06.html#e2013-06-10T12_43_46.txt</link>
<title>Google Ocean bathymetry update - World Oceans Day</title>
<dc:date>2013-06-10T12:43:46-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[Saturday (2 days ago) was <a href="http://worldoceansday.org/">World Oceans Day</a>.
We made a small G+ post for it:
<br /><br />
Google Earth: <a href="https://plus.google.com/u/0/+GoogleEarth/posts/2L5VA93uvnP">Revealing the Ocean on World Oceans Day!</a><br/>
Google Maps: <a href="https://plus.google.com/u/0/+GoogleMaps/posts/6tMAhQRAfuP">On World Oceans Day, it's the perfect time to dive in and explore!</a>
<br /><br />
<img width="600" height="419" title="Google Ocean for World Oceans Day" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/google-ocean-plus-post-world-ocean-day.png"/>
<br /><br />
The text from the post and the image captions:
<pre>Here's a sneak preview of our work to improve our ocean map in
Google Earth and Maps in partnership with NOAA's National Geophysical
Data Center (<a href="http://goo.gl/zZpPB">http://goo.gl/zZpPB</a>)
and the University of Colorado CIRES program (<a
href="http://goo.gl/b3dGH">http://goo.gl/b3dGH</a>).
<!-- -->
Explore more today by downloading this Google Earth tour list <a
href="http://goo.gl/jyDanHere">http://goo.gl/jyDanHere</a>
<!-- -->
Visit a detailed ocean landscape in Boundary Bay, WA. NOAA's Crescent
Moegling says, "Within the Strait of Georgia, deep draft vessels
transit to several large refinery facilities. It's our job to provide
the mariner with the most accurate chart, with the most relevant
features and depths available for their safe navigation."
<!-- -->
Notice the new high resolution 16 meter NOAA survey update (right)
relative to the lower 1 km resolution across the Canadian border
(left) in this area west of Bellingham, Washington.</pre>
The kmz showing the locations of the updates is here:
<br /><br />
<a href="http://commondatastorage.googleapis.com/bathymetry/kml/NOAA-NGDC-OceanMapFootprints.kmz">http://commondatastorage.googleapis.com/bathymetry/kml/NOAA-NGDC-OceanMapFootprints.kmz</a>
<br /><br />
Many thanks for the over 1000 +1's for the two G+ posts!]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-06.html#e2013-06-07T21_31_57.txt</link>
<title>NMEA Transport, Annotate and Group (TAG) block encoding</title>
<dc:date>2013-06-07T21:31:57-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[Before I start, there are so many strange things in the NMEA specification,
but this is one of my favorites:
<pre>Since there is no provision for guaranteed delivery of messages and
only limited error checking capability, this standard should be used
with caution in all safety applications.</pre>
Wahoo!  And off we go with safety of life applications without additional thought.
<br /><br />
I finally started looking at the NMEA TAG block specification.  I paid
for version 4.0 of the spec, so I should use it, right?  I have to say
it is slightly painful.   I'll do my best to describe what TAG block
is, the issues that it is trying to address, and how well the authors
did in their design.  I really hate how standards often do not list
the authors / contributors.  No way to start up a discussion.
<br /><br />
<b>The motivation for a format change.</b>
<br /><br />
The basic NMEA 0183 (3.0 and lower) sentence is missing some things.  First, it is
limited to 79 characters per line.  Therefore, there are many times
that one line is too short for all that you would like to send.  AIS
introduced the concept of messages that span multiple lines with
fields to help re-assemble the total message.  This falls apart if
you have a single feed coming from multiple devices that may all be
doing the same thing giving intermixed multi-line messages.  If we
can identify the source of the message, then we can re-assemble these
multi-line messages without fear of mixing unrelated parts.
<br /><br />
The next issue is the lack of metadata in the NMEA0183 message format.
Unless the message has a timestamp built in, there is no standard way
to log the time.  We may have additional metadata requirements and
there is no place to put them.
<br /><br />
<b>History</b>
<br /><br />
The USCG created an initial metadata format somewhere in the pre-2005
time frame that appended fields after the existing message.  I've
documented that format elsewhere, but to summarize, it consists of
comma separated fields that start with a letter code except the last
field, which is the unix UTC timestamp.  It works and that is what
I've used since 2006 for my logging and processing in software line
ais-py, noaadata-py and libais.
<br /><br />
I created a python regular expression that parses the basic structure of these
"old USCG format" messages.  Here is an example message and the regular expression
being used to pull apart the fields.
<pre><i><font color="#9A1900">#!/usr/bin/env python</font></i>
<!-- -->
<b><font color="#000080">import</font></b> re
<!-- -->
nmea_re <font color="#990000">=</font> re<font color="#990000">.</font><b><font color="#000000">compile</font></b> <font color="#990000">(</font>r<font color="#FF0000">'''^!(?P&lt;talker&gt;AI)(?P&lt;string_type&gt;VD[MO])</font>
<font color="#FF0000">,(?P&lt;total&gt;\d?)</font>
<font color="#FF0000">,(?P&lt;sen_num&gt;\d?)</font>
<font color="#FF0000">,(?P&lt;seq_id&gt;[0-9]?)</font>
<font color="#FF0000">,(?P&lt;chan&gt;[AB])</font>
<font color="#FF0000">,(?P&lt;body&gt;[;:=@a-zA-Z0-9&lt;&gt;\?\'\`]*)</font>
<font color="#FF0000">,(?P&lt;fill_bits&gt;\d)\*(?P&lt;checksum&gt;[0-9A-F][0-9A-F])</font>
<font color="#FF0000">(</font>
<font color="#FF0000">  (,S(?P&lt;slot&gt;\d*))</font>
<font color="#FF0000">  | (,s(?P&lt;s_rssi&gt;\d*))</font>
<font color="#FF0000">  | (,d(?P&lt;signal_strength&gt;[-0-9]*))</font>
<font color="#FF0000">  | (,t(?P&lt;t_recver_hhmmss&gt;(?P&lt;t_hour&gt;\d\d)(?P&lt;t_min&gt;\d\d)(?P&lt;t_sec&gt;\d\d.\d*)))</font>
<font color="#FF0000">  | (,T(?P&lt;time_of_arrival&gt;[^,]*))</font>
<font color="#FF0000">  | (,x(?P&lt;x_station_counter&gt;[0-9]*))</font>
<font color="#FF0000">  | (,(?P&lt;station&gt;(?P&lt;station_type&gt;[rbB])[a-zA-Z0-9_]*))</font>
<font color="#FF0000">)*</font>
<font color="#FF0000">,(?P&lt;time_stamp&gt;\d+([.]\d+)?)?'''</font><font color="#990000">,</font> re<font color="#990000">.</font>VERBOSE<font color="#990000">)</font>
<!-- -->
match <font color="#990000">=</font> nmea_re<font color="#990000">.</font><b><font color="#000000">match</font></b><font color="#990000">(</font><font color="#FF0000">'!AIVDM,1,1,,A,14eG&gt;3@01kqiIs8ICROownFn0D03,0*02,d-106,S0993,t140726.00,T26.49646933,r09STWO1,1370786847'</font><font color="#990000">)</font>
<!-- -->
msg <font color="#990000">=</font> match<font color="#990000">.</font><b><font color="#000000">groupdict</font></b><font color="#990000">()</font>
<b><font color="#0000FF">for</font></b> key<font color="#990000">,</font> val <b><font color="#0000FF">in</font></b> msg<font color="#990000">.</font><b><font color="#000000">iteritems</font></b><font color="#990000">():</font>
    <b><font color="#0000FF">print</font></b> <font color="#FF0000">'%s: %s'</font> <font color="#990000">%</font> <font color="#990000">(</font>key<font color="#990000">,</font> val<font color="#990000">)</font>
<!-- -->
<i><font color="#9A1900"># result:</font></i>
<i><font color="#9A1900">'''</font></i>
<i><font color="#9A1900">body: 14eG&gt;3@01kqiIs8ICROownFn0D03</font></i>
<i><font color="#9A1900">slot: 0993</font></i>
<i><font color="#9A1900">t_sec: 26.00</font></i>
<i><font color="#9A1900">t_min: 07</font></i>
<i><font color="#9A1900">station_type: r</font></i>
<i><font color="#9A1900">seq_id: </font></i>
<i><font color="#9A1900">t_hour: 14</font></i>
<i><font color="#9A1900">chan: A</font></i>
<i><font color="#9A1900">string_type: VDM</font></i>
<i><font color="#9A1900">fill_bits: 0</font></i>
<i><font color="#9A1900">sen_num: 1</font></i>
<i><font color="#9A1900">s_rssi: None</font></i>
<i><font color="#9A1900">t_recver_hhmmss: 140726.00</font></i>
<i><font color="#9A1900">station: r09STWO1</font></i>
<i><font color="#9A1900">time_of_arrival: 26.49646933</font></i>
<i><font color="#9A1900">talker: AI</font></i>
<i><font color="#9A1900">checksum: 02</font></i>
<i><font color="#9A1900">time_stamp: 1370786847</font></i>
<i><font color="#9A1900">x_station_counter: None</font></i>
<i><font color="#9A1900">total: 1</font></i>
<i><font color="#9A1900">signal_strength: -106</font></i>
<i><font color="#9A1900">'''</font></i></pre>
Remember that you can use <a href="http://kodos.sourceforge.net/">kodos</a> to try out python regular expressions.
<br /><br />
<b>TAG block</b>
<br /><br />
Apparently, that style was not good enough.  So some group of people
tried to create something better.  I'm going to ignore the control aspect.
I typically only deal with logs from devices that are out of my influence.  
We get what we get.
<br /><br />
The basic structure of TAG is to put additional data in the front of
the original NMEA 0183 messages between two "\" characters.  This
is extra interesting in that most programming languages use the back-slash
to start escape codes (e.g. "\n" and "\r"; c.f. "man ascii" and "man printf").
<br /><br />
An example to show what this looks like.  I got this chunk back in 2009.
<pre>\g:1-2-73874,n:157036,s:r003669945,c:1241544035*4A\!AIVDM,1,1,,B,15N4cJ`005Jrek0H@9n`DW5608EP,0*13
\g:2-2-73874,n:157037*1D\$ARVSI,r003669945,,172036.69698935,1376,-095,0*15
\g:1-2-73875,n:157038,s:r003669945,c:1241544035*45\!AIVDM,1,1,,B,15NEcU0001JriI4H@2DEN038069@,0*00
\g:2-2-73875,n:157039*12\$ARVSI,r003669945,,172036.77711928,1379,-097,0*1B
\g:1-2-5624390,n:2281546,s:r003669959,c:1241544037*7D\!AIVDM,1,1,,A,15PlLL@P1@JmA=DGWcw9M?w:069@,0*1D
\g:2-2-5624390,n:2281547*25\$ARVSI,r003669959,,246060,9999,-094,0*34
\g:1-2-5624393,n:2281555,s:r003669959,c:1241544037*7C\!AIVDM,1,1,,A,15PlLL0OhgJn2ORG`TQ6nEC:28ES,0*3F
\g:2-2-5624393,n:2281556*26\$ARVSI,r003669959,,246060,9999,-077,0*39
\g:1-2-5624394,n:2281557,s:r003669959,c:1241544037*79\!AIVDM,1,1,,A,18KKL00025rsc0&lt;G&lt;pMbI8E80@ET,0*73
\g:2-2-5624394,n:2281558*2F\$ARVSI,r003669959,,246060,9999,-112,0*3B
\g:1-2-5624396,n:2281561,s:r003669959,c:1241544037*7E\!AIVDM,1,1,,B,181:JhP025Jl`t8Fo0U3c2q80&lt;01,0*02
\g:2-2-5624396,n:2281562*24\$ARVSI,r003669959,,246060,9999,-117,0*3E
\g:1-2-5624398,n:2281566,s:r003669959,c:1241544037*77\!AIVDM,1,1,,A,13:4&lt;:002CJniU4G;qe:MHG40@E`,0*00
\g:2-2-5624398,n:2281567*2F\$ARVSI,r003669959,,246060,9999,-114,0*3D
\g:1-2-5624421,n:2281618,s:r003669959,c:1241544037*78\!AIVDM,1,1,,A,15N9wuUPAJJnNGrGV&gt;8J983:0D04,0*23
\g:2-2-5624421,n:2281619*20\$ARVSI,r003669959,,246060,9999,-053,0*3F
\g:1-3-60450,n:131065,s:r003669946,c:1241544038*4B\!AIVDM,2,1,9,A,55MwpAh000000000000&lt;OCO;GF2222222222220k1H4,0*2C
\g:2-3-60450,n:131066*10\!AIVDM,2,2,9,A,3140002P00000000000000000000,2*79
\g:3-3-60450,n:131067*10\$ARVSI,r003669946,9,172038.11029899,1429,-091,0*2C
\g:1-3-60451,n:131068,s:r003669946,c:1241544038*47\!AIVDM,2,1,0,B,55MwpAh000000000000&lt;OCO;GF2222222222220k1H4,0*26
\g:2-3-60451,n:131069*1E\!AIVDM,2,2,0,B,3140002P00000000000000000000,2*73</pre>
Pretty overwhelming and hard to read.  Let's start off by creating and
decoding a message that gives the Unix UTC timestamp, which is indicated by a 'c:' character.
We won't give a NMEA 0183 msg to go with it.  My reading is also that the checksum within
a TAG block ( "\\[^/+]\\" ) is optional, but I will start off with including the checksum.
<pre><tt><i><font color="#9A1900">#!/usr/bin/env python</font></i>
<!-- -->
<b><font color="#000080">from</font></b> operator <b><font color="#000080">import</font></b> xor
<b><font color="#000080">import</font></b> time
<!-- -->
<b><font color="#0000FF">def</font></b> <b><font color="#000000">checksum</font></b><font color="#990000">(</font>sentence<font color="#990000">):</font>
  <b><font color="#0000FF">return</font></b> <font color="#990000">(</font><font color="#FF0000">'%02x'</font> <font color="#990000">%</font> <b><font color="#000000">reduce</font></b><font color="#990000">(</font>xor<font color="#990000">,</font> <b><font color="#000000">map</font></b><font color="#990000">(</font>ord<font color="#990000">,</font> sentence<font color="#990000">.</font><b><font color="#000000">split</font></b><font color="#990000">(</font><font color="#FF0000">'*'</font><font color="#990000">)[</font><font color="#993399">0</font><font color="#990000">][</font><font color="#993399">1</font><font color="#990000">:]))).</font><b><font color="#000000">upper</font></b><font color="#990000">()</font>
<!-- -->
timestamp <font color="#990000">=</font> <b><font color="#000000">int</font></b><font color="#990000">(</font>time<font color="#990000">.</font><b><font color="#000000">time</font></b><font color="#990000">())</font>
<i><font color="#9A1900"># 137079172</font></i>
<!-- -->
body <font color="#990000">=</font> <font color="#FF0000">'c:%d'</font> <font color="#990000">%</font> timestamp
msg <font color="#990000">=</font> <font color="#FF0000">'\\{body}*{checksum}\\'</font>
<!-- -->
<b><font color="#0000FF">print</font></b> msg<font color="#990000">.</font><b><font color="#000000">format</font></b><font color="#990000">(</font>body<font color="#990000">=</font>body<font color="#990000">,</font> checksum<font color="#990000">=</font><b><font color="#000000">checksum</font></b><font color="#990000">(</font>body<font color="#990000">))</font>
<i><font color="#9A1900"># \c:1370791724*31\</font></i></tt></pre>
We now need to start creating the regular expression to parse these TAG things.
Let's start without worrying about the internal fields.  I came up with
<pre>\\.*\*(?P&lt;checksum&gt;[0-9A-F]{2})?\\.*</pre>
The back-slashes need to be escaped, so a single back-slash gets written as "\\" in the regex string.
Now we can add the section for the timestamp.  Sadly, the spec asserts that the timestamp is an integer.  They then
add the twist that the value can be either seconds or milliseconds, but does not give us an indicator for which.  Ouch.
Not like they had run out of letter codes.  Currently, timestamps have 10 digits and TAG block started in 2008.
<pre>int(1e9)
# 1000000000
<!-- -->
datetime.datetime.utcfromtimestamp(1e9)
# datetime.datetime(2001, 9, 9, 1, 46, 40)</pre>
Therefore, we need our integer to have 10 or more digits for the timestamp.  I came up with this regex:
<pre>\\c:(?P<timestamp>\d{10,15})\*(?P<checksum>[0-9A-F]{2})?\\.*</pre>
Viewed in kodos:
<br /><br />
<img width="600" height="411" title="Unix UTC timestamp" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-06/kodos-nmea-tag-block-unix-time.png"/>
<br /><br />
So we have a start!  More in a future post.]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-05.html#e2013-05-25T21_58_11.txt</link>
<title>Behind the scenes at Google IO</title>
<dc:date>2013-05-25T21:58:11-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[Google I/O was a totally insane experience for me this year.  I got a
substantial amount of my sleep for the week on mass transit or a bean
bag in the speaker prep area.  I know a lot of the teams are working
hard on sharing more of the how it was done.  I wanted to share a
couple more images and some more of what I've been up to now that I
have had a chance to sleep some.  
<br /><br />
First some general team news.  I've moved offices this week.  Before,
I was sitting with the amazing imagery ingest team that handles bring
in and processing all that amazing satellite and oblique imagery along
with the terrain.  I am still working with them, but I am now sitting
with the Geo Outreach and Earth Engine Teams.  For the first time
since joining Google full time, I get to sit next to the other two
full time Ocean team members.  Woot!  At Google, we do an amazing
amount over VC, but there is nothing like just being around the team.
We're never going to be able to get all the 20%ers who contribute to
Oceans to be physically located in the same area.  We're just too
diverse a bunch.
<br /><br />
And now, two pictures from IO.  The first was the lineup outside
Moscone West the day before the conference started.  I was over from
the SF office to do a "tech check" of the room.  We tried out the
slides on the actual projectors and made sure we were all comfortable
with the on stage setup. 
<br /><br />
<img width="600" height="300" title="" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-05/io-lineup.jpg"/>
<br /><br />
The cloud team was deploying hundreds of sensor datalogging notes
throughout the three floors of the conference area.  I've never seen
so many sensor nodes before.
<br /><br />
<img width="600" height="450" title="Sensor Motes" withgrayborder="True" src="http://schwehr.org/blog/attachments/2013-05/io-sensor-motes-behind-the-scenes.jpg"/>]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-05.html#e2013-05-21T10_44_14.txt</link>
<title>Processing and visualizing {ship/sensor} data - the matched set</title>
<dc:date>2013-05-21T10:44:14-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[In case it wasn't clear, I'd like to re-iterate: These talks are not
meant to be about ships.  We want to you to think about YOUR data when
watching these videos.  If your data is AIS, then that's a lucky
match, but we don't expect it.  We want YOU to go forth, process lots
of data and build amazing visualizations and analyitics systems that
meet your needs (be they local or global).  We think that AIS ship
tracking makes a great example use case with many interesting stories
that can be told with the data that give us chances to demonstrate
techniques and strategies.
<br /><br />
Brendan Kenny: "Google I/O 2013 - Google Maps + HTML5 + Spatial Data Visualization: A Love Story"
<br /><br />
<iframe width="560" height="315" src="http://www.youtube.com/embed/aZJnI6hxr-c"
 frameborder="0" allowfullscreen></iframe>
<br /><br />
Francesc Campoy, Kurt Schwehr and Mano Marks: "Google I/O 2013 - All the Ships in the World: Visualizing Data with Google Cloud and Maps"
<br /><br />
<iframe width="560" height="315" src="http://www.youtube.com/embed/MT7cd4M9vzs" frameborder="0" allowfullscreen></iframe>
<br /><br />
Jenifer Austin Folkes - Google I/O 2013 - Dive Into Underwater Street View
<br /><br />
<iframe width="640" height="360" src="http://www.youtube.com/embed/6HkW0dn7vdI" frameborder="0" allowfullscreen></iframe>
<br /><br />
Amy Unruh and Kim Cameron: "Behind the Data Sensing Lab - Gathering, Processing and Analyzing Data at Scale using the Google Cloud Platform"
<br /><br />
<iframe width="560" height="315" src="http://www.youtube.com/embed/JuaBy3e6fd4" frameborder="0" allowfullscreen></iframe>]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-05.html#e2013-05-20T10_38_48.txt</link>
<title>Massimo di Stefano - Ipython Notebook for GIS</title>
<dc:date>2013-05-20T10:38:48-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[<iframe src="http://player.vimeo.com/video/53094837" width="500" height="281" frameborder="0" webkitAllowFullScreen mozallowfullscreen allowFullScreen></iframe> <p><a href="http://vimeo.com/53094837">Lightning Talk - Massimo di Stefano - Ipython Notebook</a> from <a href="http://vimeo.com/pydata">PyData</a> on <a href="http://vimeo.com">Vimeo</a>.</p>]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-05.html#e2013-05-20T10_25_39.txt</link>
<title>UK Hydrographic Office 5 minute video</title>
<dc:date>2013-05-20T10:25:39-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[Thanks to <a href="http://blog.geogarage.com/2013/05/hydrographic-survey.html">GeoGarage</a> for linking to this 5 minute video about the UK HO.
<br /><br />
<iframe width="640" height="360" src="http://www.youtube.com/embed/uW0VOZ4SRHo" frameborder="0" allowfullscreen></iframe>]]></description>
</item>
<item>
<link>http://schwehr.org/blog/archives/2013-05.html#e2013-05-18T13_22_01.txt</link>
<title>Cat nap</title>
<dc:date>2013-05-18T13:22:01-04:00</dc:date>
<dc:creator>Kurt</dc:creator>

<description><![CDATA[Cleaning out the email this morning, I ran into older images from our web cam.
The rough life of our cat... we'd had him for 8 months at this point.  Terrible
quality with this webcam, but it was cheap and uses wifi.
<br /><br />
<iframe width="480" height="360" src="http://www.youtube.com/embed/bLzt4XHvLnA" frameborder="0" allowfullscreen></iframe>]]></description>
</item>
</channel>
</rss>
