Kurt's Weblog

In case it wasn't clear, I'd like to re-iterate: These talks are not meant to be about ships. We want to you to think about YOUR data when watching these videos. If your data is AIS, then that's a lucky match, but we don't expect it. We want YOU to go forth, process lots of data and build amazing visualizations and analyitics systems that meet your needs (be they local or global). We think that AIS ship tracking makes a great example use case with many interesting stories that can be told with the data that give us chances to demonstrate techniques and strategies.

Brendan Kenny: "Google I/O 2013 - Google Maps + HTML5 + Spatial Data Visualization: A Love Story"



Francesc Campoy, Kurt Schwehr and Mano Marks: "Google I/O 2013 - All the Ships in the World: Visualizing Data with Google Cloud and Maps"



Jenifer Austin Folkes - Google I/O 2013 - Dive Into Underwater Street View



Amy Unruh and Kim Cameron: "Behind the Data Sensing Lab - Gathering, Processing and Analyzing Data at Scale using the Google Cloud Platform"

Lightning Talk - Massimo di Stefano - Ipython Notebook from PyData on Vimeo.

Thanks to GeoGarage for linking to this 5 minute video about the UK HO.

Cleaning out the email this morning, I ran into older images from our web cam. The rough life of our cat... we'd had him for 8 months at this point. Terrible quality with this webcam, but it was cheap and uses wifi.

We are still in the process of getting the talk videos out, but here is a start. I'll flush out this list as the related videos are published.

Google I/O 2013 - All the Ships in the World: Visualizing Data with Google Cloud and Maps



Google Cloud Platform - Google App Engine, Google Compute Engine, Google Cloud Storage, Google BigQuery, Google Cloud SQL and our many APIs (Maps, Terrain and many more).

Related Maps:

• Not yet available [Google Maps + HTML5 + Spatial Data Visualization: A Love Story]
• Google I/O 2013: Re-Imagining the Map (short 2 minute video)
• Google I/O 2013 - Dive Into Underwater Street View
• Google I/O 2013 - Fireside Chat with the Google Maps Team
• Google Maps Developers Live: Ships, Polylines, Symbols, Oh My! [6 months before Google I/O 2013]
• Google I/O 2013 - Project Ground Truth: Accurate Maps Via Algorithms and Elbow Grease
• Google I/O 2013 - Design Patterns for Maps - Architecture
• Google I/O 2013 - Design Principles for Maps
• Google Developers Live at I/O 2013 - Maps Developer Sandbox
• Google I/O 2013 - Making Location Meaningful with the Google Maps APIs

Related Cloud:

• Google I/O 2013 - Behind Data Sensing Lab - Gathering, Processing, Analyzing Data using Google Cloud
• Google Developers Live at I/O 2013 - Cloud
• Google I/O 2013 - Importing Large Data Sets into Google Cloud Storage
• Google I/O 2013 - Supercharge Your Google Compute Engine App with Persistent Disk
• Google I/O 2013 - Google Cloud Messaging

I would like to thanks SpaceQuest / Analyze and NOAA SBNMS for providing such exciting data for the demonstrations!

SpaceQuest / Analyze



NOAA Stellwagen Bank National Marine Sanctuary (SBNMS)

Our talks on ships at Google I/O are tomorrow.

To go with Google I/O and our Cloud Infrastructure talk "All the ships in the world," I've added oauth2client, uritemplate, gflags, and google-api-python-client to fink on Mac OSX 10.7/10.8. This lets you use OAuth2 to access Google REST APIs.

For example:

from apiclient.discovery import build
import apiclient.errors

from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage
from oauth2client.tools import run

storage = Storage('taskqueue_oauth.dat')
credentials = storage.get()
if not credentials or credentials.invalid:
  flow = flow_from_clientsecrets('client_secrets.json',
      scope='https://www.googleapis.com/auth/taskqueue',
      message='can not find client secrets')
  credentials = run(flow, storage)
http = credentials.authorize(httplib2.Http())
service = build('taskqueue', 'v1beta2', http=http)
tasks = service.tasks()

print tasks.list(project='s~my-appeng-project', taskqueue='myqueue')

You can now install bigquery and gsutil (Google Cloud Storage) tools from fink.

fink selfupdate
fink install bigquery-py27 gsutil-py27

There were a number of packages that I had to add to fink behind the scenes and I updated boto.

fink list -i oauth2client gsutil bigquery crcmod google-api-python boto socksipy google-apputil uritemplate

 i   bigquery-py27           2.0.12-1          BigQuery database access
 i   boto-py27               2.9.2-1           Python interface to Amazon Web Services
 i   crcmod-py27             1.7-1             Cyclic Redundancy Check
 i   google-api-python-clie  1.1-2             Access Google APIs via oauth2 and discovery
 i   google-apputils-py27    0.3.0-1           Google command line utility helpers
 i   gsutil-py27             3.29-2            Google cloud storage access
 i   oauth2client-py27       1.1-1             Client library for OAuth 2.0
 i   socksipy-py27           1.02-1            TCP interface to SOCKS4, SOCKS5 or HTTP proxy
 i   uritemplate-py27        0.5.2-1           RFC6570 up to level 4

/sw/bin/bq version
This is BigQuery CLI v2.0.12

/sw/bin/gsutil version
gsutil version 3.29

The San Francisco fire department has their fire boat parked right in front of the Google SF offices. I had to wait for afternoon light, so I could capture a decent image.

I don't know anything about the effeciency of these hybrids, but they have an interesting look with their wind and solar power setups. This image is from a trip several of us made to Angel Island and Alcatraz last month.

I've been doing a photo stream of ships in leadup for Google I/O this week. My blog takes substantially longer to post to than to just use G+, so I've neglected the blog. I'll try to catch up today.

Here is a picture from the 6th floor of the Google SF office of a small work boat heading under the SF Bay Bridge. I snapped this picture when I got to the office, but it's too bright to work near the windows with the morning light reflecting off the bay.

Via GeoGarage, this is a pretty good article about pilots taking ships through tough areas. Pilots or no-pilots, accidents do happen.

If you've wondered why my blogging has decrease for the last while, here is one of several of the reasons. I've been working hard on material for Google I/O. I am super excited that we will be sharing examples of how to work with geospatial ocean data at Google IO. We built some demonstrations using AIS data show of quite a few of the cloud services that Google offers. I look forward to hearing how the community uses these features and what kinds of data they are working with!

Not going to be at the conference? Check out Google I/O Extended events worldwide

• Day 1 - Mano Marks, Luke Mahe: The Other Mobile Mapping Platform: Getting Your Maps into iOS
• Day 2 - Brendan Kenny Google Maps + HTML5 + Spatial Data Visualization: A Love Story
• Day 2 - Francesc Campoy Flores, Mano Marks, and Kurt Schwehr: All the Ships in the World: Visualizing Data with Google Cloud and Maps
• Day 2 - Jenifer Austin Foulkes: Dive Into Underwater Street View
• Day 2 - Avnish Bhatnagar: Off the Grid: Going Offline with Maps and Earth

Update 3 hours later: I impressed by the interaction around these articles that have happened. Sean has been too kind in mentioning some of the material I pointed him to in the article. Rapid 7 folks have been working to clarify what they meant by their writing.

Thanks to Brendan Kenny, I just saw this: Sean Gallagher at ars technica wrote Good Morning, Captain: open IP ports let anyone track ships on Internet In 12hrs, researchers log more than 2GB of data on ships due to Automatic ID Systems.

This article is so totally wrong with it's conclusions. The issues are open ports that could possibly be DOS'ed (denial of serviced) or exploited/pwned. AIS is a broadcast technology meant for public consumption.

Note: emphasis added.

For many of the ships, the vessel's name was included in the
broadcast data pulled from the receivers. For others, the
identification numbers broadcast by their beacons are easily found on
the Internet. By sifting through the data, the researchers were able
to plot the location of individual ships. "Considering that a lot of
military, law enforcement, cargoes, and passenger ships do broadcast
their positions, we feel that this is a security risk," Guarnieri
wrote.

Guarnieri needs to do a little more research before making statements like this. Even if you plugged all these holes, AIS ship tracking data is considered open data and would still be generally available through many sources for prices ranging from free to expensive. Military vessels that are transmitting in the clear either intend to let people know or are to dumb to make sure they are either in listen only mode or turn on blue force encryption (that that still allows direction finding of them).

According to a USCG Rear Admiral Brian Salerno in official documentation from back in 2009, which I received directly from the USCG:

As a broadcast system (where communications are intended to be
received by the public), there is no expectation of privacy of any
transmitted position, binary, or safety related messages, or any
information transmitted on AIS."

Rapid 7's analysis gets closer to the mark with this:

A quick fingerprint shows that many of the systems identified
have open telnet shells, web interfaces, and VNC servers, and many of
these are connected to old and vulnerable versions of Windows.

Being pwned is generally bad, but if people are relying on this data, then doing a DOS or injecting / removing data could do unpleasant things that range from implying false cargo movements that might impact prices to adding confusion to situation awareness increasing the chances of accidents or drawing attention from bad behavior.

See also: AIS Security and Integrity post by me back in Nov 2012 for more concepts.

Update 2013-May-04: Related links... Geo Garage on 2013-May-01: Loose blips sink ships: leaky communications threaten marine vessels. My response is "ARG!". The article at TechNewsDaily is frustrating. This stuff is supposted to be decodable and usable by the public:

To suggest that most seafaring ships âÄî including tankers, fishing
boats and military vessels âÄî could be hacked would be an insult to
industrious hackers everywhere.

Instead, reading a ship's private or sensitive communications requires
no hacking knowledge whatsoever.  The amount of publicly broadcast,
potentially sensitive material on the ocean is staggering.

Yes, I realize that it doesn't say that this is bad or anything new, but it tries to lead the reader in that direction. Geogarage at least links to more opinions on both sides of the argument.

I am entertained by his pointer back to his post from May 2005:

Google can track ships at sea ; detailed maps planned of sea bottom

If you want some fun, check out 23 minutes into this video by MTJ and then watch our upcoming Google I/O talk where we will explain how you can track ships (or build all sorts of other tools on Google cloud infrastructure)...

Some parts of oauth 2.0 have just worked for me. However, there are parts that I just don't seem to get. Eventually, we'll be sharing a whole sets of tools working together with oauth2 being the authentication glue for the whole thing. But right now, I've just watched this:



and just got these two bootks: Getting Started with OAuth 2.0 and OAuth 2.0: The Definitive Guide; Practical Information for Building Clients and Servers

Can't wait to have this all figured out.

Check out this video of Trey describing TrailScribe. Trey does an awesome job of describing the idea and I really like the illustrations!

This sounds pretty much exactly what I was trying to advocate for during 1993-96 when I was doing more field work on land. I would add a couple features to Trey's design. First would be a VHF or UHF data link with repeater to pass around the team members' positions, sample report positions and maybe any text they entered. If you initially drop a repeater or two (maybe with a basic weather station and GPS base station) on high points in the field area, then the team could have pretty good sharing during the day. Even areas that do have cell coverage usually totally loose it in areas. Second would be to use Glass or something like it to allow the science team to snag images easier. It might also take low res context picture every few minutes (would be good to let people know when that image was going to be taken for privacy). You'd still want a camera with a macro feature for a lot of photos. Finally, it would be great to have speech-to-text to give a first past of transcribing audio notes. Science vocabulary is notoriously difficult, but rough starting text would be awesome.

I spent a lot of time thinking about this while mapping the Southern Snake Range. During they day, we were in 2 person teams mapping the structure (unit boundaries, strike and dips, etc). We often had evening discussions about what we were seeing between groups as we were transfering our field notes from our field map to the group map at camp.

Today, Trey made a post about the Best Talks at PyCon US 2013. I didn't make it to pycon (other than stopping by a sprint session for an hour), so this is definitely awesome. The first video that I watched was this:

Log Analysis with Pandas [Taavi's web notes]

Video Link



Next on my list is:

Building an image processing pipeline with Python



I will be following that up with:

Building full-stack scientific applications in Python

IPython and especially the Notebook feature have been awesome additions to my python toolbox. I think that Fernando more than deserves this award... more than a decade in the making.

Fernando Perez, Winner of the 2013 Free Software Foundation Award

Sadly I was out sick and have to catch Matt Deans' talk via youtube. Matt and I worked together at NASA Ames back in the 90's in what then called the Intelligent Mechanisms Group (IMG) and is now called the Intelligent Robotics Group (IRG).



Also, in a frustrating moment, I was talking to someone at the NSF Earth Cube workshop last week at Scripps Institution of Oceanography and they were asking me about Google's tools for education. For some reason I didn't write down which one of the attendies asked me that and Philip's name escaped me during the short time of the conversation. So in the hopes that who ever that was reads my blog, please check out Philip Guo's Google Research page. Philip indroduced me to Course Builder:

Helping the World to Teach [Google Research Blog] by Peter Norvig about Course Builder.

I have been talking about a github like community around databases. Thanks to Dustin Mitchell for pointing to the city of Chicago's posting of their databases literally to GiHub.

https://github.com/Chicago/osd-pedway-routes/tree/master/data

Back when I was working with Rob Braswell and Michele Jacobi back 5 or 6 years ago, I really didn't think that the "Portsmouth Reponse" web app that we were working on would be come the big deal that is now called ERMA. Pretty amazing.

From Paper to Pixels: Mapping Pollution Response in the Digital Age [NOAA'S RESPONSE AND RESTORATION BLOG]