06.17.2014 13:39

Argo Floats

Earlier this year, Michelle made a super cool vidoe about Argos Floats.

The floats are popular and there are other videos out there.

Posted by Kurt | Permalink

06.17.2014 10:35

Looking at large diffs

Here is a little Unix command line trick. I've been looking at the differences between version releases in software like GDAL, HDF4, HDF5, etc. There is often a lot of noise (aka non-functional changes) that get in the way of my trying to understand what is different. The diff tool is pretty useful and I often look at diffs in emacs and appreciate the highlighting of Diff mode. However, I'd rather not have this noise. In the case of GDAL, that is a lot of copyright and svn Id changes. Those don't change the bevior of the code. Here is how to get them out using the diff ----ignore-matching-lines (-I) flag. Just repeat the flag for each thing you want to not get used for diffs. e.g.
diff -ruN gdal-1.1[01].0/port -I 'Copyright' -I '$Id:' > port.diff
That says to recursively produce "unified" format diffs with the contents of new files shown, but leave out lines with "Copyright" and "$Id:'. The two -I flags drop the diff file from 7500 lines to 6200 lines for the changes between GDAL 1.10.0 and 1.11.0 in the port directory. Every little bit of help makes things easier.

Posted by Kurt | Permalink

06.12.2014 08:35

US fishing and passenger exemption for greater than 65 feet

I didn't realize that this exemption is in the USCG AIS Carriage Requirements (emphasis added):
- 164.46 Automatic Identification System (AIS).

    (a) The following vessels must have a properly installed,
    operational, type approved AIS as of the date specified:

     (1) Self-propelled vessels of 65 feet or more in length, other
     than passenger and fishing vessels, in commercial service and on
     an international voyage, not later than December 31, 2004.
So, if you were a 1000 ft fishing ship in US waters, you don't need AIS. Not that there are any fishing ships that large. Any why passenger vessels? It also looks like passenger vessels also get a pass, but if they enter a VTS area and have 151 or more passengers then they have to have AIS. So weird. These rules are so poorly written. Why in the world do we still have the dates in the rules when those starting dates are 10 years ago? I guess the USCG has no concept of version control.

AIS REQUIREMENTS - Title 33, Code of Federal Regulations - 164.46 Automatic Identification System (AIS).

Posted by Kurt | Permalink

06.11.2014 19:44

Global Ship tracking on the Google Cloud at Ocean Agenda

Posted by Kurt | Permalink

06.11.2014 09:34


Today is the start of a multi-day Interagency Ocean Observation Committee - Data Management and Communications Steering Team (IOOC DMAC) meeting. Sadly, I've always been a remote attendee as the rep from Google. I've been struggling with the question of how do we all, meaning the entire community, and especially the one actor I can control - my self, have the greatest positive impact on the future of the ocean sensing / observing community. It's a challenging question. All participants were asked a bunch of questions beforehand.

Sometimes people feel that they are not some big powerful organization and that they have no influence on things. As a grad student and as a professional, I have felt the weight of bureaucracy and the in-elasticity of people have been doing things one way for years. What can each individual do? Turns out that any one person can do quite a lot. Here are some thoughts and I'll try to make them actionable.

  • Try to stay positive in interactions with others. Technology and science can be frustrating and confusing. Try not to let that creep into interactions with others. If someone flames you, do not flame back. Especially when doing things like writing a bug report or a response to a bug report. On the other hand, do not let that keep you from being direct.
  • Report bugs in any software or hardware that you use (be it open or closed). Strive to write clear bug reports that make it as easy as possible to figure out what version of things you are using, what are the symptoms, and if you think there might be a solution, try to explain what it might be. Include a picture, screenshot, code, or data sample whenever possible. I've gotten lots of "It doesn't work" bug reports and then I have to ask back, "What does not work and how is it not working. Whenever possible, have these bug discussions out in the public where others can learn from it. If search engines can't find the info, then there is little value in it.
  • Learn to program and read open source code. Maybe you never will feel comfortable submitting a bug fix back to a project, but hopefully you will. Even if it is a spelling fix in a comment, your help is immensely important.
  • When writing code (even small couple line programs), use a style guide and write clean code. How you create data and write code is just like formal technical writing. You might be annoyed by some of the things in those guides, but they are in there because other people have long suffered with bad code and have specific reasons for almost all of those entries in the guides. e.g. While it may be cool to have single letter/symbol variables and operators in the land of mathematics, but that does not work well in software.
  • Really dig into taking notes as you work, put more energy into writing up the methods for you use, and get them out to the world. So few papers out there (including my own) have enough info that someone else could get the same results. It's hard, but we have to strive for it.
  • Stick to open data formats and standards whenever possible. Design by committee sucks, but design by yourself in a closet is just as bad. If you have to make something new, get others to review it who are domain experts before committing to strategy. You don't have to take there advise, but you should ponder what they are saying.
  • Be willing to admit mistakes and errors. If we all were perfect, then we wouldn't be humans. Improvement and learning go best when we use mistakes and errors for learning. Think constructive criticism and avoid the destructive type.
  • Learn from everyone. Everyone is accustomed to newer people learning from experienced people, but it really is a two way street. People who are new may find a different and possibly better way. Or the trouble they have may inform the experienced folk as to what needs to be changed or better explained.
That's enough for now. I'm on the call as I finish writing this and thought it was great to hear someone talk about file format advice. For that person, I have this as another data point: toils-of-ais

Posted by Kurt | Permalink

05.26.2014 19:39

Confusion over SRT and ExactEarth's press release

I posted this yesterday on google plus.

TL;DR Probably one of: it's NOT AIS (using some other freq/comm encoding), it's an out of spec Class B AIS, or a spamming binary message with EEC to compensate for expected BER. I strongly dislike that they do not say.

I'm going to have to call BS on this press release from exactEarth. They do not say what exactly their technology is in a ground based class B transceiver is that makes it better for being able to be received by satellite. If they are truly altering how they transmit class B and somehow embedding more information, then these AIS Class B units are out of spec and you should expect to have them confiscated if you use them in any countries waters (even if it's a mode that flips on/off). My best guess is that they are using a frequency owned by someone else and transmit on that with some other system. So... it's not ITU 1371.1-4 AIS at that point. Lots of other people do tracking in other frequency ranges and do that via satellites. A quick example... SPOT. I really dislike announcements like this that are all unsubstantiated hype. I don't expect system definitions in a press release, but they could reference docs frequency allocations, tx methods and power.

Or am I missing something huge? Are they fiddling with the encoding, power shaping each slot datagram, phase shifting the transmission to increase SN/correlation?

How do they get this extra information in? That implies that they are using an un-approved message type. Or they could be using a Application Specific Message (ASM)/Binary Broadcast Message(BBM) defined by some country? Then they might just spam the channel with lots of really small packets at high transmit rate. The obvious solution is to use Msg 6, 8, 26 or 27 and just fill the channel. If you ignore the AIS checksum info and have the payload contain really robust error correction data, you don't have to worry about trying to get a correct decode. Just save the data from all the known time slots and merge recoverable chunks until you have the full message. If I knew that a particular AIS unit was going to transmit the same message N times starting at know times and each TX had error correction built into the bit stream, I could definitely get the chances of receiving them message to near 100% for each satellite pass. You combine that with phase discrimination (those ships infront of the sat are shift up in freq and those behind are lower in freq), and you have a pretty decent receive rate. Being out in international waters, maybe you can get away with stuff like this. Try this in US waters and you should expect to have a visit from the feds if you are in the AIS frequencies (assuming that they can get their tracking act together). This is what a lot of other communications protocols have done for ages. It's sad that the creators of AIS never put any of this into the AIS core spec. But maybe we should be glad they didn't, because if it mirror the way other AIS specs were written, it would be an unimplementable disaster.

There are a few other possibilities, but those are the most likely.


Additionally quotes like this make no sense to me. TDMA doesn't make the job of tracking a particular ship easier, but it in no way prevents you from receiving a non-Class A (e.g. Class B) message.
the TDMA radio access scheme which makes AIS such a robust
system, means that satellite detection of terrestrial transmissions
has been limited only to Class A transceivers
And if 95% of the AIS transceivers out there use SRT as their core, we have a monopoly. I need to take a look at the Class B vender field to see what the distribution is these days.

I argue that we should avoid proprietary solutions and without more info, it appears that the ABSEA thing is likely very much a closed proprietary system. I look forward to SRT publishing more information and and open (and patent-free) standard that all can implement (both tx & rx). We really should have a message for the high seas. The minimalistic AIS satellite message isn't great. Would be so much better to have a standard message for the high seas (low load VDL areas) that has EEC and transmits on a known slot. e.g. We could have a hash that converts an MMSI to a particular start slot. And if you know that a number of repeats will give the same exact message, you really have something to work with.

The current receiver design seems fundamentally flawed in requiring all messages to pass the checksum reporting. I wish that receivers would optionally report bad messages with a flag detailing the failure.

Posted by Kurt | Permalink

04.05.2014 08:24

More open sensor format discussion

Originally posted here: https://github.com/tkurki/navgauge/issues/4

Kees Verruijt pinged me discussion and suggested I should add my $0.02 on requirements. Interesting discussion. For me, JSON is nice to have on the side, but I can't really consider a text based message as the primary format. I've been thinking about how to replace things like Generic Sensor Format (GSF) and the zillion vender binary formats for logging multibeam and sidescan sonars, LIDAR (e.g. tons of work in the PulseWave project), high rate intertial navigaton systems (IMU/INS), the full working set of what can be expressed for smaller sensors with NMEA 0183 messages, AIS, Radars, imagery, carrier phase GPS (aka RTK, Rinex format,...), acoustic recoders (hydrophones and geophones), e&m, an open version of MISLE ship incidents, whale sitings, data license, software versions used, the whole metadata world (ISO paywalled disaster), etc. And then think about all this combined for a very large number of platforms... e.g. all AIS receivers in a network, all buoys, AUV, ASV, etc in a fleet. On some systems, either because high resolution timing is key or because they are underwater for long periods of time without GPS (AUVs or dataloggers that may sit on the sea floor for months), multiple time sources may be essential. And I would love to see more at the OpenROV, OpenCTD, Arduino/Raspberry Pi level for cheaper and smaller sensors. We then as a community need the ability to conflate data over long time series. e.g. Maritime Rule X changed on July 1st, 2007... how did the env change in the years before and after? Did that increase or decrease regional fuel use, ship incidents, whale strikes, ship noise, economic activity at local ports, etc? I'm already see streams that are GB to TB per day and need to scale up from there. For the browser, JSON is great for small things, but the overhead of strings to binary for rendering of say just 10M points in Chrome, forced a switch to binary over the wire.

Even just the ARGOS float project has the potential to explode... they only have a few sensors right now, but what happens when they get tons more sensors and multiple routes for data to make it to a usable datastore.

I've been worrying about this for a long time now. I saw the issue with the number of sensors on the Healy all trying to spew NMEA 0183-ish stuff to NAIS with the USCG to working with sonars. I tried to start writing some of my ideas here: http://schwehr.org/blog/archives/2014-02.html#e2014-02-18T09_06_12.txt (and other blog posts). However, I haven't gotten very far. I tried making a message definition language of my own to support AIS, but did not get very far in the RTCM community. My current thought is that the best core definition place to start is with Protocol Buffers (ProtoBufs) and RecordIO or similar. That would allow for other serializations, but start off with a well defined message content that has a build-in binary serialization combined with the ability to add other serializations (e.g. JSON, XML, proto ascii, etc) to the mix fairly easily. I'm also realizing that there is a need for a journalling style system (append only) that could allow for index messages to be written to log files that would make access to logs much faster. This becomes especially important if you are later trying to join chunks of logs that might come anywhere from near realtime to months after the fact. There are plenty of other issues in there... e.g. nothing seems to talk about recording the calibration process in sensor log streams or things like time from which source and how accurate is it? And what license is the data released under? The bathymetry attributed grid (BAG) format contains ISO XML metadata, but they only have "classified" / "non-classified" for data. What about public domain, proprietary, creative commons license X, etc. ?

I think any spec has got to be totally open and it would be best to be machine readable and usable for all for any purpose. e.g. paywalled or GPL are a real problem. Something like the Apache 2 license seem like the only way to go if we really want to open up data interchange.

I've spent a lot of time trying to survive the existing disaster of standards and want to be moving towards a world in which data is easier and more fun to work with from small devices to global scale fleets of sensors. Hopefully some of these thoughts are useful to the discussion you all are having here.

Some links to material that might be interesting for you all:

P.S. GSF is now homed at Leidos (which SAIC spun off) here: https://www.leidos.com/maritime/gsf (and still not yet under an open source license).

Posted by Kurt | Permalink

04.03.2014 10:33

Disabling autologin on a mac from the command line

Just did this remotely on a mac. I had historically left autologin running. This was blocking some system admin tasks. Here is what I did to kick myself out from the commandline on an ancient install of Mac OSX 10.6. 10.9 likely has things the same.

how-to-disable-automatic-log-in-via-the-command-line-in-os-x/ and what-is-the-mac-os-x-terminal-command-to-log-out-the-current-user
ssh myhost
ps -aux | grep $USER | wc -l
defaults read  /Library/Preferences/com.apple.loginwindow autoLoginUser
sudo defaults delete /Library/Preferences/com.apple.loginwindow autoLoginUser
sudo osascript -e 'tell application "System Events" to log out'
ps -Ajc | grep loginwindow | awk '{print $2}' | sudo xargs kill
ps aux | grep $USER | wc -l

Posted by Kurt | Permalink

03.19.2014 19:08

Rebecca Moore at the White House - Google Earth Engine

Posted by Kurt | Permalink

03.17.2014 10:45

AIS talkers for NMEA VDM/VDO messages

I got an email with this question:
In my data feed m getting some sentences with packet types I do not
recognize.  Could you tell me what they mean?


I'm also getting multi-sentences with different packet types. Is that
allowed? First part can be ABVDM and second part AIVDM.
I was just asked in email to explain the first field in the CSV that is the AIS NMEA armored messages. Sadly, the NMEA specification is paywalled by the annoying folks at NMEA so I can't share with you where talkers are defined. The same goes for the IEC standards that define AIS device messages and testing procedures. And I only have draft versions of a few of those that people on the commitees have given me over the years. For example, I have "Maritime navigation and radiocommunication equipment and systems - Digital interfaces - Part 100: Single talker and multiple listeners - Extra requirements to IEC 61162-1 for the UAIS".

Back before ITU 1371 was opened up, I helped Eric Raymond (ESR) a bit with his writing of AIVDM/AIVDO protocol decoding. I still think that his document is drastically better written than the crap that is in the ITU documents. ESR also has NMEA Revealed with a list of talker ID.

Let's start with a sample message:
The first two letters after the bang ("!") are the "talker." In NMEA speak, this is the device that generated the text in your local environment. It has nothing to do with any remote AIS devices and what they sent over the VHF radio. Above, we have "AI". This is the classic "AIS" device talker. It is what you get with most AIS receivers and transceivers (they are NOT transponders).

Following the talker are 3 letters that define the "sentence type." In the world of NMEA, this is the type of message that you are about to parse. VDO is a type that not many people know about. It says that the message that follows is an AIS message that is about "own ship." This really means that the device is talking about itself. The more commonly observed VDM means "AIS message that the device received over a VHF radio channel." The VDO in the above sample message makes sense as this is an AIS basestation telling us the message 4 (basestation report) for itself. The message has two extra fields on the right: ",b003669730,1249053583". This is the older USCG logging format and has a receiver identifier and UNIX UTC timestamp. The b at the beginning of the receiver name says that the message was logged from a basestation. If we decode that message, we should find that the MMSI of the basestation report should match the receiver name.

This next sample message has "BSVDM". The talker is "BS". This means that we are hearing from a basestation. The logger has added ",r003669717" with an "r" for receiver, but it might just be missconfigured. This message is from my 2006 notes and I was just figuring out AIS and what the USCG was doing back then. I did have a basestation on my desk for a while.
And a final sample that has "ANVDM". This was an AIS ATON (aid-to-navigation) device that uses "AN" as the talker:
As for "AB", I was unsure. I did receive some sample data with an AB talker, but I do not know what kind of device was doing the logging. AB might stand for Ais Basestation in someones mind. Looking at NMEA's closed documentation (yes, I paid rediculous amount for version 4.0 of the spec), I found:
AB Independent AIS Base Station
AD Dependent AIS Base Station 
AI Mobile Class A or B AIS Station
AN AIS Aids to Navigation Station
AR AIS Receiving Station
AS Limited Base Station
AT AIS Transmitting Station
AX AIS Simplex Repeater Station
SA Physical Shore AIS Station
There was no "BS" talker in there. Don't ask me to explain the differences between all those codes.

Please, please, please: If you have the opportunity to talk to people at NMEA, IMO, IALA, IHO, ITU, or IEC, tell them that closed / paywalled specifications are the work of the devil (and contradictory to the mission of SOLAS). It's bad enough that people aren't willing to put the discussion leading up to these specs online with their names attached. Can we please at least get ALL specs for these public data standards (because they are mandated by law in many cases) out there for free distribution licenses?!?!?

Posted by Kurt | Permalink

03.06.2014 21:43

Software Defined Radios

Last time I went looking for Software Defined Radios (SDR) to play with, the price was in the $300-$500 and up. That is quite different than some of the options we have today. I just ordered at NooElec NESDR Mini SDR & DVB-T USB Stick (R820T) w/ Antenna and Remote Control. It's a R820T tuner IC made by Rafael Micro with a frequency range of ~ 25MHz to 1750MHz. It apparently can output 2.5M S/s (million samples per second).

Posted by Kurt | Permalink

02.24.2014 00:12


I had never really used pkg-config before today. Turns out to be reasonably simple. I read through the man page an here is me messing with it.
type -a pkg-config
pkg-config is /sw/bin/pkg-config

dpkg -S /sw/bin/pkg-config
pkgconfig: /sw/bin/pkg-config

fink list -i pkgconfig | grep -v virtual
 i  pkgconfig                0.25-2  Manager for library compile/link flags

pkg-config --help
pkg-config --list-all
pkg-config --list-all | wc -l
pkg-config --list-all | head -4
gstreamer-tag-0.10 GStreamer Tag Library - Tag base classes and helper functions
libmetalink libmetalink - Metalink library
xf86rushproto XF86RushProto - XF86Rush extension headers
fontsproto FontsProto - Fonts extension headers

pkg-config x11 && echo 'yes'

pkg-config missing && echo 'yes'
pkg-config missing || echo 'do not have "missing"'
do not have "missing"

pkg-config --print-variables x11

pkg-config --print-provides x11
x11 = 1.6.2

pkg-config --print-requires x11

pkg-config --cflags x11

pkg-config --libs x11
-L/opt/X11/lib -lX11

Posted by Kurt | Permalink

02.18.2014 09:06

A unified marine sensor logging format

This is the first of a series of posts I hope to write.


See also: Toils of AIS by Eric S. Raymond and Kurt Schwehr.

I am frustrated that each sub-community of marine related sensors has their own data formats for raw logging. Even worse, each vendor has their own format (with many different messages as they innovate over the years). Those formats require a wide range of support libraries and often not easily extensible. We have formats for multibeam sonars, seismic systems, acoustic recorders (hydro and geo-phones), seismic stations (really just a 3-axis, 3-channel geo-phone), lidar, ships' and GPS sensors, magnetometers, radar, grids, images, vectors, metadata, and more. Those formats treat data in other formats as orthogonal. Most of encodings used for those formats are not even described in the Wikipedia article on Comparison of data serialization formats. Worse yet, some key standards are behind paywalls, which greatly hampers adoption (e.g. try pricing all the ISO standards related to metadata). On top of that, some of these standards are so confusing that it is nearly impossible to create conformant data streams.

In this series, I will be arguing that the sonar community needs to get away from systems like the Generic Sensor Format (gsf) and move to a unified machine and human readable specification system for the data stream at the lower levels. I will be pushing for using ProtoBuffers (protobuf). Protobufs were created by Google and are heavily used within Google. I have a fair bit of experience with them as I have been working for Google for the last 2 years. However, there are other formats, such as BSON, that could work. I do feel like I'm repeating myself with this. I argued for a custom XML representation of AIS messages starting in 2006 and declaring defeat in 2010 as the AIS community opted to stick with their amazingly terrible status quo.

An additional concern is that people have proposed formats or tools that are either only used in a small community, are not tested across a number of key programming languages or revert to defining the raw bit level packaging such that the software engineers of every piece of code that will touch this format will have to understand the low level details.

In this first post, I'd like to explore the use cases. What are the range of problems that we are trying to solve? I'm attempting to think about 50 or so years into the future, a near impossible task, but some things will most certainly be correct. Storage of data will become more affordable in terms of cost, power required, volume/mass required. Compute power for an available budget will increase. Sensors will become more affordable and their diversity will increase.

I have not had a chance to flush out these use cases, but I do not want to wait any longer to get this starter post on line.

* Single beam echo sounder from a ship

* Multibeam echo sounder ship

* Multi-channel seismic vessel

* Lidar from an aircraft

* An AUV, Glider, Argos style device with IMU

* Super cheap land sensor networks without GPS

* ADCP mounted on the sea floor

- without external intervention for long periods - event based time corrections

* An array of sea floor devices

- down for an extended period with slow intermittent acoustic connections

* Ping editing a multibeam sonar

* Post processing to get time and location

* Post collection indexing to allow fast access

* Later splicing of multiple platforms into a single stream

- inlining of forward and backwards index messages

* Non-survey ship underway

Posted by Kurt | Permalink

02.07.2014 10:50

Google Docs Mac OSX 10.9 Mavericks

For my desktop, I have linux (I gave up on macs for the desktop), but for my mobile devices, I've been using Mac devices. Mac OSX is really frustrating at a low level compared to Linux, but I like the hardware (minus the lack of easily fixing). Here is what I do to survive in the Mac OS X world. I've given up Microsoft Office, Photoshop and Illustrator now that I'm not a full time professor making posters and living in the painful UNH IT environment. I used to do this in MediaWiki format inside of CCOM, but I don't want to have to tunnel into CCOM for this and I'd like it to get more eyeballs. I rarely got feedback from others, so why would I keep it closed? I will be leaving out stuff specific to the places I work in a lot of cases, but I will share a bit when it doesn't expose confidential business practices.

At the time of this post, the document is hardly even a Work-In-Progress (WIP).

Mac OSX 10.9 Setup

Posted by Kurt | Permalink

02.07.2014 10:17

AutoAwesome winter in NH

AutoAwesome on G+ is often just cheesy, but I really like this one.

Posted by Kurt | Permalink

01.23.2014 06:03

Cross country flight

I just recently got to fly from West Coast to East Coast (US that is). I always enjoy trying to capture good pictures when I have access to a window. A few years back, I always seemed to get people sitting next to me that claimed there was no way to get good photos from a commercial flight. Thankfully that has dropped off.

Images taken with an iPhone 4S. The color is never quite up to what I see with the eye and iPhoto's color tools are not good enough to do much of a fix.

The Sierras should be totally blanketed in snow by late January. This photo scares the heck out of me. California is in for a serious drought this year.

The Southern Snake Range / Great Basin National Park:

It's very sad to see hydrolic fracking pads all over the place. I only recently learned to look for. Ignore the water treatment on the right and look for the mostly bare earth rectangles. What are we doing to the lands that we have to live and grow food on and the aquafers that provide drinking water? Short term gain (for some) and long term disaster (and we don't even know what all trouble we are in for). See also: frack.skytruth.org

Posted by Kurt | Permalink

01.01.2014 09:30

Starting out 2014

I'm not as cool as people like pydanny with posts like New Year's Meme 2013 that cover What's the Coolest Thing You Discovered This Year, What New Programming Technique Did You Learn This Year, Which Open Source Project Did You Contribute to the Most This Year? What Did You Do, What Are the Top Three Things You Want to Learn Next Year, and What is the Top Thing You Wish Someone Would Write Next Year?

I just try to survive the onslaught. As a part of that I have some rituals. The most important one is that I keep a logfile where I track what I do every day in emacs org mode. I have a top level ('*') entry for each day and '**' entries for each general thing I work on during the day. For each day, I usually start with a '** todo :todo' section. I write in there all the things that I hope to get done that day in the form of a org todo list in the form of '- [ ] fix bug foo'. I'm always overly ambitious and rarely get them all done. In fact some of them require more than that day or are out and out not in the range of things I can get done. I have a 'todo' alias that greps the current year for all lines that start as '- [ ]' (or contain *+ TODO). I often get to the point in the day where I'm totally overloaded with the current task and just staring at the screen. That's a good time for a task swap. So I run my todo alias and look through the back log trying to pick off a few small things to keep me going until I'm ready to get back into the current fight. That list gets long as the year moves on.

The best part of the new year is that I make a new org mode file and get to start with nothing on the todo list. I then skim last year's list and pull forward only those things that still feel like they should be in my conciousness. Those rafted todos are still important, but they don't deserve to cloud my mind. They serve to document what I was trying to do, things I thought would be nice, etc. They also provide a pool of potential side projects for others. Google has the concept of the 20% side project and I sometimes get asked by people how they can help. If I need inspriration, I can look back through past years of todo items. As always, I have to thank Anne Wright for kicking me into the one log per year mode (and away from being dominated by un-grep-able paper log books) back in 2004 in MER mission control during a middle of the night image processing hack session with the Ames crew.

The other thing I do is to reset my shell logs. I log every command that I type on the command line on my work laptop and desktop (that returns successfully). That's about 50k commands each. It logs when, what git branch I was in, what was the current directory, and the command. Admittedly, this is pretty boring as it is dominated by ls, cd, egrep, find, locate, emacsclient, git status, git diff, and so forth. But, with that many commands, there are quite a few gems and I grep for every day. At the New Years, I move the current log from .shell.log to .shell.log.YEAR.

Top shell commands for 2013:
  1. ls 13289
  2. cd 9170
  3. git 5162
  4. grep 2349
  5. rm 2170
  6. find 1855
  7. less 1722
  8. fink 1312
  9. schwehr 1209 # Alias to ssh into my desktop
  10. ipython 1199
  11. mv 985
  12. open 953
  13. svn 949
  14. branch 848 # Alias to create a git branch
  15. ssh 795
  16. du 764
  17. make 720
  18. cp 713
  19. pwd 712
  20. ec 695
  21. mkdir 664
  22. e 637
  23. df 633
  24. echo 602
  25. scp 601
  26. gcutil 498
  27. gdalinfo 496
  28. cat 486
  29. tar 462
  30. bq 433
  31. gsutil 424
  32. fg 406
  33. python 406
  34. bg 406
  35. type 396
  36. sudo 394
After using the cut command to pick out the field with the command, I used this little bit of python to print the summary:
#!/usr/bin/env python

import collections
import os

cnt = collections.Counter()

for line in open('cmds'):
  cmd = os.path.split(line.strip())[1]
  cnt[cmd] += 1

for cmd, num in cnt.most_common(40):
  print '<li>' + cmd, str(num) + '</li>'

Posted by Kurt | Permalink

12.24.2013 16:30

Hello go (golang)

I've been sick for 10 days and am going totally stir crazy. I figured it was time to finally get over the initial kickstart of using go. Till now, I've always left the direct use of go to my collaborator Francesc. I've read his code and it performs nicely. So it's time for me to try out go in my own way. Thanks to Brendan, there was an initial fink go package for version 1.1.1. However, this didn't work for me on Mac OSX 10.9. I tried fixing it, and bumped into the oddities of the go src package. Some go authors refer to this as the go way. I consider this just having a weak source setup process and no install method. Once it was clear that go is just like python with the system checking the timestamps against the compiled libraries (.go -> .a for go; .py -> .pyc for python), I was able to build a go 1.2 package that even includes bash command line completion (which still acts a bit strangely) and the emacs go mode. So being that I'm a fink maintainer, let's start there.
fink install go

echo "(require 'go-mode-load)" >> ~/.emacs

type go  # /sw/bin/go

go version
# go version go1.2 darwin/amd64

go help  # Lots of command options on the screen

mkdir -p go/src/hello  # Violate the go tree layout for now cause we don't need it yet.

cd go/src/hello
cat > hello.go << EOF
package main

import "fmt"

func main() {
	fmt.Printf("Hello, world.\n")

go run hello.go
# Hello, world.
go build hello
can't load package: package hello: cannot find package "hello" in any of:
	/sw/lib/go/src/pkg/hello (from $GOROOT)
	($GOPATH not set)
So that was not really the right way to setup your go projects, but it was easy enough to get a kickstart. How about something a little more interesting? I like regular expressions (regexp) and want to see how it would be to write a NMEA parser in go. I got stuck right off, but I got help over on stack overflow: Using named matches from Go regex. I really prefer to use named fields for regular expressions. It's slower, but the regular expressions are much more understandable.
mkdir ~/go/src/regexp

cd !:1  # !:1 says use the argument at position 1 in the prior shell command

emacs demo_regexp.go
Then put this code into emacs:
package main

import (

var myExp = regexp.MustCompile(`(?P\d+)\.(\d+).(?P\d+)`)

func main() {
  match  := myExp.FindStringSubmatch("1234.5678.9")
  result := make(map[string]string)
  for i, name := range myExp.SubexpNames() {
     result[name] = match[i]
  fmt.Printf("by name: %s %s\n", result["first"], result["second"])
go run demo_regexp.go 
# by name: 1234 9
Now to change it up, I wanted to try go via Google App Engine (GAE). Google App Engine in Go. The download is a little bit confusing. I expected an installer or something. What I found was a zip that you have to unpack and put in your path. I guess I should make a fink package in my personal package tree for that, but some other day.
fink install wget unzip
wget http://googleappengine.googlecode.com/files/go_appengine_sdk_darwin_amd64-1.8.8.zip
unzip go_appengine_sdk_darwin_amd64-1.8.8.zip
cd go_appengine
export PATH=`pwd`:$PATH
mkdir -p ~/go/myapp/hello  
cd ~/go/myapp/hello  
cat << EOF > hello.go
package hello

import (

func init() {
    http.HandleFunc("/", handler)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprint(w, "Hello, world!")
cd ~/go
goapp serve myapp &
open http://localhost:8080
I know I do things differently than the average person, I'm not on the Go team, and this is definitely not a sanctioned tutorial, but it's how I got started. I have tons more to do.

Posted by Kurt | Permalink

12.13.2013 17:58

Gebco multibeam cookbook

I was talking to people at AGU about common issues in multibeam sonar data. I made a comment that there is not great documentation out there that goes through the common issues seen in multibeam covering what the issues look like, what causes them, can they be prevented, and can they be fixed after the fact? Someone (perhaps Dave Sandwell) asked if I had seen the Gebco Cookbook. I hadn't.
6.2.4 Common multibeam swath sonar errors
Spikes under nadir
Tracking water-column noise
Roll-over on inward-facing steep slopes
Anomalies on outward-facing slopes
Navigation missing on first ping
Inaccurate sound speed profile used
Here is one of those:
Tracking water-column noise 
Modern multibeam swath sonar systems generally include automated
bottom tracking algorithms. Such algorithms search for seafloor echoes
in a window centered on the previous (or adjacent) ping’s
depths. However, the system can begin to track water column noise, and
may continue to do so if the system is not actively monitored. Depth
variability between pings can be large, the across-track depth
profiles are not consistent with seafloor topography, and most of the
beams/pings should be discarded.
There aren't any images and the text is pretty short. So now I've seen it, but it seems pretty weak. This stuff is definitely a start, but it's so far from what I'd like to see in the end. And the work is copyrighted in a way that I'm not going to contribute to it: "© Copyright International Hydrographic Organization [2012] " It would be nice if were at least some sort of open documentation license.

Posted by Kurt | Permalink

11.22.2013 12:40

NextGov Bold award for Whale Alert

Congrats to Dave Wiley and everyone who made Whale Alert possible. We are quite the large, distributed and non-traditional team.

Nextgov Honors Feds for Innovative Tech That Saves Time, Money and Lives
A NASA hackathon, an app to track whales and an initiative to allow
federal employees to use their own smartphones and gadgets for work
were among projects recognized Wednesday with the first annual Bold

Nextgov received nearly 200 nominations for the contest. Our editorial
staff selected 19 finalists -- individuals and teams from a dozen
agencies -- that exemplify the kind of creative problem-solving,
technical acumen, ambition and persistence we frequently hear about in
the private sector but too seldom learn about in government.
There were 7 winners, including:
David Wiley of the National Oceanic and Atmospheric
Administration led a team to produce Whale Alert, a mobile app to
prevent endangered right whales from colliding with ships.
Monica pointed out a minor note to the above description. It would really be better stated that this app is for preventing ships from colliding with the right whales.

What Politicians Don't Tell You About Federal Employees - Nextgov Bold Awards Finalists

NOAA's Stellwagen Bank National Marine Sanctuary created Whale Alert,
which provides up-to-date information about the critically endangered
North Atlantic right whale, displaying data on nautical charts via
iPad or iPhone platforms. To protect right whales from strikes shipthe
leading cause death ofNOAA has promulgated a suite of rules and
voluntary measures that affect East Coast shipping. Prior to Whale
Alert, information was delivered to ships in piecemeal fashion, making
compliance difficult and resulting in industry fines. With more than
17,000 downloads, the app has been highly successful improving
conservation efforts.

Posted by Kurt | Permalink