Kurt's Weblog

I want to be able to run some Django code from a cron job to update a database. I looked at Django Cron, but then thought about the admin commands and realized that was a better route to my goal. Here is the basic route that I tool:

django-admin.py startproject projname
cd projname
python manage.py startapp myapp
# Edit settings.py to add myapp to the INSTALLED_APPS
cd myapp
mkdir -p management/commands
touch management/__init__.py
touch management/commands/__init__.py
cd management/commands/

Now create your command:

import django.core.management.base as base

class Command(base.NoArgsCommand):
  help = 'Describe the Command Here'
  def handle_noargs(self, **options):
      print 'hello world'
      print options

Now give it a try:

cd ../../.. # back to the project directory
python manage.py help



Type 'manage.py help ' for help on a specific subcommand.



Available subcommands:
  cleanup
  compilemessages
  ...
  flush
  hello_world
  inspectdb
  loaddata
  ...
python manage.py hello_world
  hello world
  {'pythonpath': None, 'verbosity': '1', 'traceback': None, 'settings': None}

That's it. Now I have a command (that doesn't really do anything yet) that I can put in my crontab.

Now if I can just figure out why deserialization goes into infinite recursion on me.

We just had Dana Yoerger from WHOI give seminar about ABE, SENTRY, and NEREUS exploring the deep. He mentioned a video of deep underwater volconism that sounded pretty exciting, so I went hunting and found this: an eruption at 4000 feet (1200 m).

CCOM has several images that were used in the strategic plan presentation this week by the President of UNH.

In this first image, the student is pointing at GeoZui4D image of EK60 and DeltaT sonars with whales visible in the mid-water multibeam data created by Roland. On the left is an image also from GeoZui, but it is hard to make out.



In this next image, the student is touching with his left hand (on our right) a risk analysis of grounding for a ship entering the Portsmouth, NH harbor done by Brian Calder.



The above images are 14 minutes into the video:

UNH in 2020 from UNH Video on Vimeo.

We've been working with Jamie at Google to get the data into Google Earth. Jamie discovered several bugs in tools that we were using to get the data out. Note that a lot of other CCOM data is already in Google Earth, but comes up as "NOAA".

Wander the seafloor like never before [Google Lat Long blog]

...
Several organizations have provided their ship-collected data for
publication in Google Earth to improve our undersea maps. The Center
for Coastal and Ocean Mapping - Joint Hydrographic Center, has shared
large swaths of underwater depth data collected from their expeditions
north of Pt. Barrow, Alaska into the Arctic. The Living Oceans Society
has shared their surveys off of the west coast of British Columbia,
Canada, so you can now zoom around the Oglala seamount
...

The announcement: GDAL/OGR 1.7.0 Released. All the details turns out to be a very long list: 1.7.0-News

The summary of changes:

• New Raster Drivers: BAG, EPSILON, Northwood/VerticalMapper, R, Rasterlite, SAGA GIS Binary, SRP (USRP/ASRP), EarthWatch .TIL, WKT Raster
• GDAL PCIDSK driver using the new PCIDSK SDK by default
• New Vector drivers : DXF, GeoRSS, GTM, PCIDSK and VFK
• New utilities: gdaldem, gdalbuildvrt now compiled by default
• Add support for Python 3.X. Compatibility with Python 2.X preserved
• Remove old-generation Python bindings.
• Significantly improved raster drivers: GeoRaster, GeoTIFF, HFA, JPEG2000 JasPer, JPEG2000 Kakadu, NITF
• Significantly improved vector drivers: CSV, KML, SQLite/SpataiLite, VRT

The most interesting to me are the new BAG and GeoRSS drivers along with the gdaldem command line program for hillshading and coloring. It will be interesting to see how the GeoRSS works and which of many flavors of GeoRSS that it outputs. SpatiaLite is still on the todo queue, but does not have a very high priority.

Fink has not been updated to this version. Make sure that you are up to date on libgeos3 (current is 3.2.0).

NOAA Signs Contract for Seven Seabat 7125-SV [Hydro International]

...
A real-time uncertainty output from the SeaBat 7125 may be used in
PDS2000, along with information from other sensors to calculate a TPE
(Total Propagated Error). Soundings may be filtered by setting either
the relevant IHO order or by defining a custom vertical error
limit. Other new features in SeaBat 7125-SV include roll
stabilization, XYZ offsets for flexible transducer installations,
Quality filter and advanced diagnostic which increase survey
efficiency. Further, AutoPilot uses sonar tuning values from a default
or user generated look-up table for hands free sonar operation.
...
Six of the SeaBat sonar systems are to be mounted on Hydrographic
Survey Launches of the SV Rainier, SV Fairweather, SV Thomas Jefferson
and SV Ferdinand R. Hassler while the seventh system will be mounted
on the SV Nancy Foster.
...

Ron Schott has been running a fantastic Outcrop/Deskcrop series on his blog where he shows off Gigapan's from the field and hand samples back in his office. Well worth following if you enjoy geology.




On the same note, Art Trembanis just sent me a link to his students' blogging of their trip to New Zealand. I've been following some of it on facebook and it looks like they have had an amazing field experience this year:

NZ GeolMast Winter 2010

From the Art Trambanis/CSHEL news channel:

A Year of Global Shipping Routes Mapped by GPS [Wired Science]

I would have guessed that they used S-AIS (satellite) messages picked up by SpaceQuest, ComDev, or OrbCom. But it turns out that they just used great circle routes from itineraries.

Credit to Dale Chayes for the details of the paper: The complex network of global cargo ship movements by Kaluza, Kolzsch, Gastner, Blasius. The Abstract:

Transportation networks play a crucial role in human mobility, the
exchange of goods, and the spread of invasive species. With 90% of
world trade carried by sea, the global network of merchant ships
provides one of the most important modes of transportation. Here we
use information about the itineraries of 16,363 cargo ships during the
year 2007 to construct a network of links between ports. We show that
the network has several features which set it apart from other
transportation networks. In particular, most ships can be classified
in three categories: bulk dry carriers, container ships and oil
tankers. These three categories do not only differ in the ships’
physical characteristics, but also in their mobility patterns and
networks. Container ships follow regularly repeating paths whereas
bulk dry carriers and oil tankers move less predictably between
ports. The network of all ship movements possesses a heavy-tailed
distribution for the connectivity of ports and for the loads
transported on the links with systematic differences between ship
types. The data analyzed in this paper improve current assumptions
based on gravity models of ship movements, an important step towards
understanding patterns of global trade and bioinvasion.

To give you a feel what an actual global data set feels like, you can interact with the global Vessel Observation System (VOS) data in Google Earth yourself by going here:

http://vislab-ccom.unh.edu/vos/ (By Ben Smith)

Or you can take a look at 2 minutes of S-AIS data from SpaceQuest over South Africa:

S-AIS-AprizeSat-20090729.kmz

When I saw this video, I had flashbacks to working with Geb Thomas and the CMU Field Robotics Center on a panospheric camera for the Nomad rover in the Atacama desert.

Our paper on the topic came out back in 1997: Operating Nomad during the Atacama Desert Trek [PDF]



Found via News: Google Earth in the Classroom, More 3D in Australia, DIY StreetView [Google Earth Blog]

Aliens are assisting in the ocean drilling project off of Antarctica...
You can follow the work on the JOIDES Resolution via Facebook.

Picture This: NOAA, Google Join Forces to Visualize Scientific Data

The agreement lists six topic areas in which NOAA and Google may
pursue cooperative research projects of mutual interest:

    * Engaging the public in ongoing and historic scientific
      expeditions including those of the NOAA ship Okeanos Explorer;
    * Compiling and improving bathymetric datasets to display in
      Google Earth and make available for downloading;
    * Expanding NOAA efforts to publish oceanographic data, especially
      data from the NOAA-led Integrated Ocean Observing System;
    * Expanding NOAA efforts to publish climate data, especially data
      from the greenhouse gas monitoring system;
    * Increasing the amount of data available for NOAA’s Science on a
      Sphere, an educational Earth science display system,
      [http://sos.noaa.gov/] by adapting it to display files in the
      Keyhole Markup Language, the file format Google Earth and Google
      Maps use for geographic data; and
    * Providing interactive access to marine zoning and regulatory
      information concerning regions such as continental shelf
      boundaries and marine protected areas.

It's a shame that there are no images in the press release.

Mano Marks from Google has posted his Slides from my upcoming talk at Berkeley

Yesterday, I took along my Spot personal locator (points available for 6 days) to give it yet another chance. When I tried it last year, the results were really bad. I still thing the interface is horrible, but this time the device did a lot better job. I got 8 position reports during 2 and a half hours while out at the LNG ship off of Boston. I wore it on my belt the whole day. It got my first morning okay message, but failed to get my second hello. Out of a 12 hours of running, I got 2.5 hours of position reports mostly while on the bridge of the ship (where I never expected it to work). We spent about 2 hours on the upper deck of a workboat, but never got a postition report from that. Overall, with a brand new battery installed that morning, the unit did better, but I still think this is not a well designed device.

Yesterday, I got the chance to go out to the LNG ship Explorer, which is currently at the North East Gateway (NEG) terminal. This was my first time on a super tanker. In the morning, I met the Gateway Endeavor in Salem.



We had a short ride out that included training on how to go through a basket transfer. You put your arms through the ropes and stand on the outside of the orange ring at the bottom. You throw your luggage luggage in the middle. Then they crane you from ship to ship. It wasn't a big deal on such a nice day, but it must be pretty crazy if you have to do that in foul weather.



This is the ship that we transfered onto. She is HUGE. She is off loading gas to the pipeline right now and providing natural gas to heat homes in New England.



We headed up to the bridge and got a look at the whole ship from up high. You can see on the left that it extends out crazy far.



Here is what the bridge looks like. The Explorer is a very new ship and bridge has a very clean design. I noticed that pilot port for AIS is right up front in the middle and has a power outlet right next to it.



Here is me on the deck of the ship next to the piping that transfers gas around the ship.



We were waiting for our crane ride off the ship to the awaiting Manisee workboat that (eventually) took us back to Salem. Apparently, in 1986, the Manisee was used in the movie One Crazy Summer when it used to be a car car ferry boat.



To finish off the day, I got a great view of the sun setting over Boston.

I didn't catch if Aaron or Dane pointed this out, but it's a great tool. GPSd has a command line tool called gpsdecode that knows about AIS as of GPSd version 2.90. Here I am using a svn trunk copy from yesterday. gpsdecode can output JSON, so importing the data to a variet of tools should be fairly easy. First some sample data from today:

grep AIVDM gall-2010-01-20 | head -10 | tail -4
!AIVDM,1,1,,A,152JPuOP00JvJ8nHvW`0j?v020Sn,0*55,d-053,S2239,t235959.00,T59.70994503,r01SSSP1,1263945603
!AIVDM,2,1,3,A,59NS7F@2=bJl7PHC:204i85:222222222222221AE`P,0*46,d-107,S2093,t235955.00,T55.82982737,r003669959,1263945597
!AIVDM,2,2,3,A,G94N@0E@DVQEp;hC1iDR@H888880,2*4E,d-107,S2093,t235955.00,T55.82982737,r003669959,1263945597
!AIVDM,1,1,,A,19NRpnh00SJq:;<GkkNm?4Al0@QF,0*34,d-108,S2096,t235955.00,T55.90987913,r003669959,1263945597

That should be a msg 1, msg 5 (that covers two parts), and a msg 1.

Then decoding... a message 14 appears that should not be there, but the shipdata msg 5 is not. Hopefully, this will give me a chance to contribute to Eric S. Raymond's (ESR) hard work adding AIS to GPSd (thanks Eric!)

grep AIVDM gall-2010-01-20 | head -10 | tail -4 | gpsdecode -j | grep class
{"class":"AIS","type":1,"repeat":0,"mmsi":338075893,"scaled":true,
 "status":"Not defined","turn":"nan","speed":0.0,"accuracy":false,
 "lon":-70.2528,"lat":43.6530,"course":200,"heading":511,"second":0,
 "maneuver":0,"raim":true,"radio":4588}
{"class":"AIS","type":14,"repeat":1,"mmsi":223232012,"scaled":true,
 "text":"L#X3NSP5MS$7CPBAR               TUZHE2QG$"}
{"class":"AIS","type":1,"repeat":0,"mmsi":636008667,"scaled":true,
 "status":"Under way using engine","turn":0,"speed":3.5,"accuracy":false,
 "lon":-71.3995,"lat":41.6100,"course":1340,"heading":136,"second":58,
 "maneuver":0,"raim":false,"radio":135340}

Here is my first program to read AIS data with GPSD. First I started the gps daemon such that I can see what it is doing in a terminal:

sudo gpsd -N -G -D 4 ais://localhost:31414

Then I created this code that does a dumb poll. This is not a good way to write a program to listen for data. It will be hard on the system.

#!/usr/bin/env python
import gps
import time

daemon = gps.gps(host='localhost', mode=gps.WATCH_ENABLE|gps.WATCH_JSON|gps.WATCH_SCALED)

while True:
    if not daemon.waiting():
        time.sleep(0.01)
        continue
    msg = None
    try:
        msg = daemon.next()
    except:
        print 'ERROR: some bug with GPSD'
        continue

    if msg['class'] != 'AIS':
        continue

    if msg['type'] in (1,2,3):
        print 'position message from: ', msg['mmsi']
        continue

    print 'msg:', msg['type']

Then when I run it, I see messages coming in!

./gpsd_test.py 
position message from:  366909530
position message from:  367043190
position message from:  366974680
position message from:  310470000
position message from:  367192060
position message from:  367360990
position message from:  367078250
msg: 4
msg: 4
msg: 14
position message from:  366998520
msg: 14
position message from:  205445000

We got two snow storms in a row with the 2nd finishing up late last night. With the sun out, NH is a magical place. This is the view out my office window today.



Taken with ye-old-iphone.

Les and I have been discussing the rolls that IRC might play at the office and on ships for at least the last 6 months. I finally did a quick test to see if I could get things running in just a few minutes. The results were great! I have a running server and we had two different users from one local account and one out on our private network. Here is the process:

fink install ngircd
sudo ngircd -n -p # Keep it forground and don't try to connect to other servers

[58376:5    0] ngircd 13-SYSLOG+ZLIB+SSL+TCPWRAP+ZEROCONF+IDENT+IRCPLUS+IPv6-powerpc/apple/darwin9.8.0 started.
[58376:6    0] Activating: no-daemon-mode, passive-mode.
[58376:6    0] Reading configuration from "/sw/etc/ngircd.conf" ...
[58376:4    0] No administrative information configured but required by RFC!
[58376:5    0] No SSL server key configured, SSL disabled.
[58376:4    0] Warning: Error during SSL initialization, continuing ...
[58376:6    0] ServerUID must not be 0, using "nobody" instead.
[58376:6    0] Running as user nobody(-2), group nobody(-2), with PID 58376.
[58376:6    0] Not running with changed root directory.
[58376:6    0] IO subsystem: kqueue (initial maxfd 100, masterfd 3)
[58376:6    0] Now listening on [0::]:6667 (socket 4).
[58376:6    0] Now listening on [0.0.0.0]:6667 (socket 5).
[58376:6    1] Successfully registered "Server Info Text" with Rendezvous.
[58376:6    1] Successfully registered "Server Info Text" with Rendezvous.
[58376:6  142] Accepted connection 6 from 192.168.8.236:60588 on socket 5.
[58376:6  142] IDENT lookup for connection 6: no result.
[58376:5  142] User "goatbar_!~goatbar_@eel.ccom.nh" registered (connection 6).

Then I fired up Converstation on the mac and added a new server. Then I added a channel to the server. Jordan then joined the channel from his machine and we have a working channel.

Interim Policy for the Sharing of Information Collected by the Coast Guard Nationwide Automatic Identification System - They are requesting comments.

SUMMARY: The Coast Guard has developed an interim policy for the access 
and sharing of information collected by the Coast Guard Nationwide 
Automatic Identification System (NAIS). The Coast Guard is also seeking 
comments on the applicability and levels of sharing of information 
collected by the NAIS, the definition of historical NAIS information, 
and any commercial or security sensitivities with respect to sharing 
NAIS information in order to assist us in the development of the final 
policy on NAIS information sharing. This policy would serve as guidance 
for Coast Guard program managers and field units regarding the sharing 
of information collected by the NAIS with foreign governments, Federal, 
State, local, and Indian tribal governments, and non-government 
entities.
...

Down aways, the document specifies their levels of info sharing:

Levels of Information Sharing

    The following three levels pertain to information collected by the 
Coast Guard NAIS.
    The First level (Level A) is unfiltered (real-time) information 
collected by the NAIS that is less than 12 hours from transmission. 
Level A information may be shared with U.S. or foreign governments for 
legitimate internal government use (i.e., law enforcement, maritime 
safety, defense, and security purposes). The final policy would clarify 
that this information should be handled in accordance with Department 
of Homeland Security policies concerning sensitive but unclassified 
information, including by marking this information ``For Official Use 
Only'' (FOUO), or any successor controlled unclassified information 
marking and handling requirements subsequently implemented by the 
Department. Level A information would be handled as FOUO, or otherwise 
in accordance with another controlled unclassified information 
designation approved by the Department, due to the potential commercial 
sensitivities of the information collected by the NAIS and the 
unfiltered, embedded addressed and encrypted information, the release 
of which may pose a security risk.
    The Second level (Level B) is filtered (real-time) information 
collected by the NAIS that is less than 12 hours from transmission. 
Level B information may be shared with foreign governments or U.S. 
Federal, State, local, and Indian tribal governments, and with non-
government entities that are contractually supporting a Federal 
government agency's operations or research and development efforts, 
Coast Guard validated port partners, or non-governmental organizations 
with which the U.S. has an established or formalized relationship 
(e.g., port authorities, pilot associations, local law enforcement 
agencies, etc.). Level B may filter out encrypted and addressed 
information as appropriate and will be filtered as the NAIS system 
filtering capabilities become available. As with Level A information, 
the final policy would clarify that this information should also be 
handled as FOUO or other appropriate designation due to the potential 
(but unverified) commercial sensitivities of the information collected 
by the NAIS and, if applicable, the embedded addressed and encrypted 
information, the release of which may pose a security risk.
    The Third level (Level C) is information collected by the NAIS that 
is more than 12 hours from transmission. This information should be 
considered historical and no longer needing to be handled as FOUO. 
Requests for filtered or unfiltered historical information would be 
processed in accordance with the Freedom of Information Act, 5 U.S.C. 
552.
    In an effort to continue to enhance navigation safety and security, 
and to protect commercial and proprietary interests, this information 
may not be used for purposes other than those intended for the 
disclosure as approved. Foreign governments, Federal, State, local and 
Indian tribal governments, and non-government entities shall not 
retransmit or redistribute the information stream in any form other 
than those intended for the disclosure as approved, shall not charge a 
fee for its usage, and will be required to execute documentation 
imposing restrictions on the use of information collected by the NAIS. 
Any provision of information collected by the NAIS to foreign 
governments will be coordinated with and through the Department of 
State, as needed.
    Implementation of the final policy would be subject to NAIS system 
capability, especially with respect to evolving capabilities to filter 
NAIS information.

Yet another video digitized from old VHS tapes. This 1998 video shows 3D models created with the Stereo Pipeline. The results are first shown in Mars Map (teal/aqua background) and then in the first version of Viz. These were done on an SGI Onyx2 Deskside and dump via a scan converter to VHS tape. The research was funded through NASA's Telepresence Intercenter Working Group (TRIWG). The last bit of the video shows level-of-detail (LOD) using mesh reduction software.