05.31.2005 19:52

segy sql database working

Just packaged up segy-py-0.3.tar.bz2 for release... sql database generation.

Doh! I just spent 15 minutes writing a nice explanation of the database building of SEGY header information (both file and trace). Then I hit the wrong key in the nanoblogger add and nuked the whole thing. Basic gist... it works. It is heavy on disk usage and cpu cycles, but I wrote it in less than an hour and a half while on the airplane. It needs better typing as I assumed everything should be saved as a 4 byte integer, but it is a great first pass.
ls -l *db


-rw-r--r-- 1 schwehr staff 18501632 May 31 19:40 bpsegy.db

sqlite bpsegy.db 'select x/3600.0,y/3600.0 from segyTrace' | head

-120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111 -120.101111111111|34.4061111111111
The sample db creation code:
#!/usr/bin/env python
from segy import Segy
import sqlite
import os


if __name__ == "__main__": dbName = 'bpsegy.db' if os.access(dbName,os.W_OK): os.remove(dbName) cx = sqlite.connect(dbName) cu = cx.cursor() xstar = Segy('bpsio01L00.xstar')

# # CREATE the database tables # cu.execute(xstar.header.sqlCreate()) if True: t = xstar.getTraceHdr(1) cu.execute(t.sqlCreate()) cx.commit()

# # POPULATE the tables # cu.execute(xstar.header.sqlInsert()) for i in range(1,xstar.getNumberOfTraces()): t = xstar.getTraceHdr(i)

if 0==i%500: x = t.getHeader('X')/3600. y = t.getHeader('Y')/3600. print i,': ',x,y cu.execute(t.sqlInsert(traceNumber=i)) cx.commit() # Flush this to the db!
My quick and dirty sql string generation code. Note that there are only INTEGER fields!
  ...
    def sqlCreate(self,specificKeys=None, table='segyFile'):
        """
        Build an SQL string to create a table for the keys/fields in a
        SEGY file header


arguments: specificKeys - in set, these are the keys that will used in the create string table - SQL table name (may have different tables for different formats) "
"" if not specificKeys: specificKeys = self.drv.binaryHeaderEntries.keys() cStr = 'CREATE TABLE '+table+' (' for key in specificKeys: cStr += ' '+key+' INTEGER,' cStr = cStr[:-1] cStr += ');' return cStr

def sqlInsert(self,specificKeys=None, table='segyFile'): """ Build an SQL string to INSERT the values for the current file """ if not specificKeys: specificKeys = self.drv.binaryHeaderEntries.keys() iStrStart = 'INSERT INTO '+table+' (' iStrEnd = ' VALUES ('; for key in specificKeys: iStrStart += key+',' iStrEnd += str(self.getBinEntry(key))+',' iStrStart = iStrStart[:-1] + ')' iStrEnd = iStrEnd[:-1] + ');' return iStrStart + iStrEnd
Feels strange to have the sql string generators inside the header instances, but that way, I know the driver is loaded.

Posted by Kurt | Permalink

05.31.2005 15:48

segy-py setup.py and distutils

I am now in the Chicago O'Hare airport. I actually found an electrical outlet next to a seat. What a rare thing in this airport. I usually end up sitting on the floor, often under a drinking fountain or pay phone.

I have been working on packaging up segy-py a bit better while on the plane, but first want to write a bit about some thoughts on python segy driver files. I keep discovering little design problems that show off the power of python. I realized one today. I was thinking about how I have to create a large python file full of dictionaries for each flavor of segy. If this were C/C++ this would definitely be the case or I could write a whole bunch of wrapper functions to go into other definitions for functionality. In python, it turns out that it should be really easy to pull parts of functionality from different config files. The example that I was thinking of was with a segy flavor that is almost segy rev1.

#!/usr/bin/env python


import segy_rev1 print segy_rev1.textFileHeaderEntries['LINE']

from segy_rev1 import textFileHeaderEntries

print textFileHeaderEntries['LINE'] textFileHeaderEntries['LINE'] = [3,5] print textFileHeaderEntries['LINE']

print segy_rev1.textFileHeaderEntries['LINE']
Running this as './segy_tweak.py' gives results like this:
  [2, 5]
  [2, 5]
  [3, 5]
  [3, 5]
This show that there is one problem or drawback to this approach. The original module data is altered. If you had two different segy files open with different drivers for each, they could step on each other, right? I did a quick check and this is a global problem (or feature if it helps you for other code). So a better way to solve this trickiness is to do a deep copy of the data.
#!/usr/bin/env python


import segy_rev1 print segy_rev1.textFileHeaderEntries['LINE']

import copy textFileHeaderEntries = copy.deepcopy(segy_rev1.textFileHeaderEntries)

textFileHeaderEntries['LINE'] = [3,5] print textFileHeaderEntries['LINE']

print segy_rev1.textFileHeaderEntries['LINE']
Which shows that the deep copy does what we want in this case:
  ./segy_tweaked3.py 
  [2, 5]
  [3, 5]
  [2, 5]  <-- The original value is not changed.  Perfect!


PACKAGING

In order for people to be able to use my segy-py package, it needs to work as a solid python package. As such, I have been trying to use the standard distutils. This started off pretty quickly, but then I ran into a few things that I figured out by looking at other packages. The first one is that I hate having to list out every source file. There ways to do this with GNU Make, but it is complicated for a beginner. With python it turns out to be simple, but I am not sure the difference between the two. I started out with something like this inside of the setup call in setup.py:
        py_modules = ['segy','ebcdic','segy_sql','ibm'],
There I have listed out each python file that I want to end up in the distributed tarball. But what I really want is every python file included. I found an example that does this:
        packages=['']
So what really is the difference between a module and a package? I bet I will discover that when I get around to trying to install the stuff with setup.py.

The next question is should I be putting my pure python sources under a subdirectory or can I leave them all in the top level right now? Will everything at the top level cause install problems? Looking in the fink directory /sw/lib/python2.4/site-packages I see a mess. There are directories, python files, .so compiled shared libraries, pth files (what are these?), and other junk. I will have to get back to this one later.

Then on to a Makefile. I almost always end up with a Makefile for everything I do. People who work on projects with me know that I love simple GNU makefiles and bash scripts. Bash is falling quickly by the wayside with python, but I have a makefile for this project. Take a quick look at part of my makefile:
PYTHON_FILES:=${wildcard *.py}
PYTHON_MODS:=${PYTHON_FILES:.py=}
PYTHON_MOD_DOCS:=${PYTHON_FILES:.py=.html}

docs: 
	@echo "${PYTHON_MODS}"
	pydoc -w ${PYTHON_MODS}
	code2html Makefile > Makefile-segy-py.html

BASE_URL:=http://schwehr.org/software/segy-py
BASE_SCP:=schwehr.org:www/software/segy-py
docs-install: docs
	scp ${PYTHON_MOD_DOCS}  Makefile-segy-py.html ${BASE_SCP}/docs
	open ${BASE_URL}/docs

help:
	open ${BASE_URL}/docs

check: test
	pychecker *.py
	grep -n FIX *.py

test:
	./segy.py -v

clean:
	rm -f *.pyc *.html *~


# dist is build by "python setup.py sdist" # and MANIFEST gets built in the process based on MANIFEST.in dist-clean: clean rm -rf dist *.db rm -rf MANIFEST

# python packaging sdist: dist-clean ./setup.py sdist --formats=bztar sdist-install: sdist scp dist/* ${BASE_SCP}/
It really does not do very much, but it helps me remember common tasks. For example, testing the code and uploading the web documentation to my server becomes as easy as this:
  make check && make test && make docs-install
Well, that does not work right now as I often have tests that fail, but you get the idea.

Why am I bringing up make other than the fact that I think make is cool? Well, the default setup.py file I had does not put the Makefile in the tarball. Doh! :(
Posted by Kurt | Permalink

05.30.2005 15:00

misc stuff

Disk recovery from crashes. The inside scoup by Toms Hardware.

WiFi security.

LinuxISOTorrent

iCluster at SIO. Many screen Mac G5 systems.

Posted by Kurt | Permalink

05.30.2005 09:22

pysqlite, python CAD, more segy-py

pysqlite 2.0.3 was just released. I need to soon switch from version 1.0!

Python CAD - no idea about his thing.

fish is a user friendly command line shell for UNIX-like operating systems such as Linux. Sounds interesting.

BRL CAD looks like an interesting CSG CAD system

I have been working on the loadable driver support for segy-py. It is getting there. I think I need to start using a real python IDE. Also, it is tough to debug problems when unit tests do not produce stack traces. The segy-py API is likely to drift a bit.

FreeAntennas.com for wifi antennas.

Posted by Kurt | Permalink

05.29.2005 09:25

first release of segy-py

I have just put together an actual release of segy-py. This is hardly even alpha code at this point. Rougher than rough. But now it has a python setup and on the web docs that get automatically updated via a Makefile. I am still learning how to make a proper setup.py, but it seems to work.
  segy-py


BTW, I was reading the MarsRoverBlog for a while, but I can not deal with the Martian alien civilization junk. Goodbye.

Posted by Kurt | Permalink

05.28.2005 12:05

python dynamic module loading

I have been wanting to implement a system for dynamically loading of driver information. I want to have a set of files where each one contains the python information needed to load SEGY files for that particular vender version. I would then have one for standard segy rev1 and possibly one for rev 0. After that, there would be drivers for xstar/Edgetech and other venders. These files would be a function to help ID the file followed by custom readers and lookup tables for what fields are where in the file. Today, I started figuring out how to load in these files. I was thinking that I would want to load the module into a "driver" namespace. Doing "import segy_xstar" is not very flexible. I want to do something like "drv = load_driver("xstar")" or "drv = load_driver("rev1")". Turns out that this is not to hard to do! Here is my first test example using the imp module.

First the file foo.py that will be loaded:
bar = 'this is in foo'
Here is main.py:
#!/usr/bin/env python
"""
Test out module loading
"""


import imp

file = open('foo.py','r') modFoo = imp.load_module('chirp',file,'foo.py',('.py','r',imp.PY_SOURCE))

print modFoo.bar
Then to run it:
  chmod +x main.py
  ./main.py


this is in foo




findsounds.com

Posted by Kurt | Permalink

05.28.2005 09:28

BitTorrent

Perhaps BitTorrent, eDonkey, or some other P2P mechanism is the real way that we should be distributing marine science data. The files are huge and it is mandatory that we have a national archive, but it new surveys were distributed via P2P, people could really get at this stuff quickly.

Python and Doxygen!

Posted by Kurt | Permalink

05.26.2005 10:02

segy python sqlite simple example

This is an example of how to quickly create an sqlite database using segy python and the sqlite module. This is nothing fancy and it is missing 99.9% of all the information available, but it is small and thus easier to understand (I hope). First the code. This is the previous example, but with a little extra sqlite code.
#!/usr/bin/env python
from segy import Segy
import sqlite


dbFilename = 'bpsegy.db' cx = sqlite.connect(dbFilename) cu = cx.cursor()

# traceCount is the trace offset in a file createStr = "CREATE TABLE traces" createStr += " ( traceCount INTEGER, lat REAL, long REAL );" cu.execute(createStr) cx.commit()

xstar = Segy('bpsio01L00.xstar')

for i in range(1,xstar.getNumberOfTraces()): traceHdr = xstar.getTraceHdr(i)

x = traceHdr.getHeader('X')/3600. y = traceHdr.getHeader('Y')/3600.

insertStr = "INSERT INTO traces" insertStr += " (traceCount, lat,long)" insertStr += " VALUES" insertStr += " (" + str(i) insertStr += " ,"+str(x) insertStr += " ,"+str(y) insertStr += " );" cu.execute(insertStr)

# Flush this to the db... cx.commit()
Once this is run, you will now have a little sqlite database in the same directory. Here is a simple query to look at a range of traces.
  sqlite bpsegy.db "select * from traces where traceCount>40 and \
  traceCount<50;" | tr '|' ' '


41 -120.101111111 34.4061111111 42 -120.101111111 34.4061111111 43 -120.101111111 34.4061111111 44 -120.101111111 34.4061111111 45 -120.101111111 34.4061111111 46 -120.101111111 34.4061111111 47 -120.101388889 34.4061111111 48 -120.101388889 34.4061111111 49 -120.101388889 34.4061111111
That is it for now.

Constantin Cranganu from Brooklyn College had an interesting over pressure poster this morning.

The Seismic Glosary is helping me get through the Sumatra talks.

CIA Factbook - Just a reminder that it is still with us.

OSS on OSX - this article covers fink too.

Windows guys gives up on MS Windows - An amusing read about someone who has had it with Windows.

Posted by Kurt | Permalink

05.26.2005 09:16

NAMSS and segy python example

NOTE: For the USGS maps, you will need a working SVG plugin. Firefox has one built in, while you will have to install one from Adobe for Safari.

I talked to Hart and Childs this morning about the National Archive of Marine Seismic Surveys or NAMSS. What really caught my eye on their poster was the Santa Barbara Basin!

From the uswest page there are a number of marked surveys in the Santa Barbara area. The red dots are not yet available, but the green are available!

Now back to segy python! I now have a working shot location dumper. I was caught off guard by the X,Y not changing in the first few traces, but I should have remembered that is the case. The traces always have the last available X,Y such that it looks like traces do not move for a while unit the next GPS data. I need to make some sort of interpolation function for this. Here is my simple trace, shotpoint, x and y dumper program.
#!/usr/bin/env python
from segy import Segy
xstar = Segy('bpsio01L00.xstar')


for i in range(1,xstar.getNumberOfTraces()): traceHdr = xstar.getTraceHdr(i)

shot = traceHdr.getHeader('Shotpoint') x = traceHdr.getHeader('X')/3600. y = traceHdr.getHeader('Y')/3600. print shot,": ",x,y

Creating a Segy option with a filename loads up the datafile as a very efficient memory map. This way the data from the file only comes into memory when needed. The hope if that if you are only reading, say, one trace, the file header and that trace are paged in, but thing else.

Then the code loops over all the traces. The range must have a "1," in it as there is no trace 0 in the SEGY specification. If you ask for trace 0, you will get garbage.

For each loop, the getTraceHdr(i), pulls out another mmap'ed reference to a particular trace. From the trace reference, I can then ask for the X,Y, and shotpoint fields. I know from prior knowledge (but you could look at the header fields too) that these X,Y pairs are in seconds of arc. To get decimal lat/long, I divide by 3600. Then I print it out. That is it! Here is the beginning and end of the output:
./segy_sql.py


1 13961172 : -120.101111111 34.4061111111 2 13961172 : -120.101111111 34.4061111111 3 13961172 : -120.101111111 34.4061111111 4 13961172 : -120.101111111 34.4061111111 5 13961172 : -120.101111111 34.4061111111 6 13961172 : -120.101111111 34.4061111111 7 13961172 : -120.101111111 34.4061111111 8 13961172 : -120.101111111 34.4061111111 9 13961172 : -120.101111111 34.4061111111 10 13961172 : -120.101111111 34.4061111111 ... 13540 13961172 : -120.1275 34.4308333333 13541 13961172 : -120.1275 34.4308333333 13542 13961172 : -120.1275 34.4308333333 13543 13961172 : -120.1275 34.4308333333 13544 13961172 : -120.1275 34.4308333333 13545 13961172 : -120.1275 34.4308333333 13546 13961172 : -120.1275 34.4308333333 13547 13961172 : -120.1275 34.4308333333 13548 13961172 : -120.1275 34.4308333333 13549 13961172 : -120.1275 34.4308333333


Other news: New Method For Imaging Dec. 26 Indian Ocean Earthquake Yields Unprecedented Results. Mentions Scripps and IGPP

Posted by Kurt | Permalink

05.26.2005 06:10


Posted by Kurt | Permalink

05.25.2005 13:58

Segy python works. Woohoo



UPDATE 17:00 - Turns out that the API below is already out of date, but I just made my first plot of a trace dumped to a sample number/value pair. Looks very much like I expected!

Talking to quite a number of people about seismic data yesterday got me motivated to really go for it with my python segy library. The task really has been mostly an exercise in making lots of lookup tables with dictionaries. Just a few minutes ago, I got my first correct lat/long pair out of a trace from an Edgetech Xstar segy file. This is extra exciting since the xstar format does not follow the SEGY standards. In fact, seismic unix (aka su or cwp-su) is unable to read xstar files. Right now, we first run them through sioseis to write out a better segy file. My hope is to have code that can easily be adapted to each vender's interpretation of the SEGY standards. Clearly people treat the SEGY documentation as suggestions. Being that I intend this to be an open source utility, here is the source. Note that this is just my test code. It is FUGLY.
  segy-1.7.py.html
Here is a little bit of pseudo code that shows how I can read from the first file/trace of the Santa Barbara data. I open up the file by creating a Segy class object and from there, I create a SegyTrace object. One problem right now is that it is possible to close the mmap from the Segy class while SegyTrace objects are still using the mmap. I do not know what would happen if that happens. Pseudo code:

	import Segy
	xstar = Segy("bpsio01L00.xstar")
        trace1 = SegyTrace(xstar.data,3600)
        trace1.printHeader()

SweepFreqStart  =  1000
RecvGrpElev  =  0
SweepLen  =  50
OverTravel  =  392
GapSize  =  235
SrcEDirTenths  =  0
LowCutFreq  =  0
...

        # look!  If we divide by 3600, we get degrees!
        x = trace1.getHeader('X'); print x, '-->  ',x/3600.
        y = trace1.getHeader('Y'); print y, '-->  ',y/3600.

-432364 -->  -120.101111111
 123862 -->    34.4061111111
Those lat/lon numbers actually look like they are in the right area. Party time! Actually, now I need to be able to seek through the traces. I have not done that yet, hence the hard coded 3600 for the trace start. Then after that, I need to build and sqlite database of the file and trace parameters.

  fugal
       adj : of or relating to or in the style of a musical fugue


fugue n 1: dissociative disorder in which a person forgets who who they are and leaves home to creates a new life; during the fugue there is no memory of the former life; after recovering there is no memory for events during the dissociative state [syn: {psychogenic fugue}] 2: a dreamlike state of altered consciousness that may last for hours or days 3: a musical form consisting of a theme repeated a fifth above or a fourth below its first statement


XML tutorials

Posted by Kurt | Permalink

05.25.2005 08:16

AGU and more

Yesterday, I talked to lots of interesting people in the OS and PP groups. I also went to the lunch meeting for the AGU informatics group. They spent their time on beaurocratic drudgery. Who is going to host the web site and what is the group to be called. Please. That is not what I went for!

MySQL cheat sheet - it is not nice to cheet.

Yafray is a raytracer that blender can use.

Verse is some sort of networked visualization/geometry system. May have some viz like aspects to it and may take the idea a lot farther, moving the visualization away from the center. The exact opposite from Viz.

PC Assembly tutorial - Intel 32-bit protected mode programming with NASM. What about PPC? Mac PCC assembly links

Beginners Guide - PowerPC Assembly Language

dumb laws

WiseGeek.com - Clear Answers for Common Questions.

Posted by Kurt | Permalink

05.23.2005 19:41

reading binary files with python

Now that I finished my poster presentation today at AGU, I wanted to take a quick look at reading segy files with python. I wanted to read straigt from mmap, but did not know how. Now I do. I have not started working with byte order, but here is the basics. Byte order can be done with Module-socket with the htonl type functions. Or better yet with the struct module with the "@=<>"

SEG publications: segy rev 0 and reb 1 are on this page.
All values in the binary File Header and the Trace Header are two's
compliment integers, either two bytes or four bytes long.  There are
no floating-point values defined in the headers.


Trace Data sample values are either two's complement integers or floating-point. This revision adds data sample formats of 8-bit integer and 32-bit IEEE floating-point. Both IBM floating-point and IEEE floating-point values are written in big-endian byte order.


Here is a little program that I wrote to tell byte order for xsonar. GCC can also tell you byte order, and I really need to write or find a couple liner in python.

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>


enum endian {LITTLE,BIG,UNKNOWN_ENDIAN}; // aka LOW, HIGH

enum endian getHostEndian(void) { int short s=1; char *b = (char *)&s; if (b[0]) return (LITTLE); return (BIG); }

int main (int argc, char *argv[]) { switch (getHostEndian()) { #if 0 case LITTLE: printf("little\n"); break; case BIG: printf("big\n"); break; #else case LITTLE: printf("-DINTEL=1 -DMOTOROLA=0\n"); break; case BIG: printf("-DINTEL=0 -DMOTOROLA=1\n"); break; #endif case UNKNOWN_ENDIAN: printf("ERROR: UNKNOWN endian\n"); exit(EXIT_FAILURE); default: printf("ERROR: unexpected endian\n"); exit(EXIT_FAILURE); }; return(EXIT_SUCCESS); }

/* NOTES - What machines return, what xsonar returns * * Darwin/Mac OSX/PPC - big * IRIX/SGI/Mips - big, MOTOROLA * Linux/x86 - little, INTEL */
Mac OSX is big endian and the same byte order as SEGY which will help performance. But I must make sure that I check out the code on an Intel machine.



First two write some binary data with a simple C++ program:
#include <unistd.h>
#include <cstdio>
#include <fcntl.h>
#include <cstdlib>


using namespace std;

int main (int argc,char *argv[]) { { int fd = open("int",O_WRONLY|O_CREAT,0644); int32_t _int32=6; write(fd,&_int32,sizeof(_int32)); close(fd); } { int fd = open("int_neg",O_WRONLY|O_CREAT,0644); int32_t _int32=-6; write(fd,&_int32,sizeof(_int32)); close(fd); } { int fd = open("float",O_WRONLY|O_CREAT,0644); float d=6.77; write(fd,&d,sizeof(float)); close(fd); } { int fd = open("float_neg",O_WRONLY|O_CREAT,0644); float d=6.78; write(fd,&d,sizeof(float)); close(fd); } return (EXIT_SUCCESS); }
#!/usr/bin/env python


import mmap import os import struct

def readInt(filename): file = open(filename,"r+") size = os.path.getsize(filename) data = mmap.mmap(file.fileno(),size) return struct.unpack("i",data[:4])[0]

def readFloat(filename): file = open(filename,"r+") size = os.path.getsize(filename) data = mmap.mmap(file.fileno(),size) return struct.unpack("f",data[:4])[0]

if __name__ == "__main__": print "testing..." x = readInt("int"); print x x = readInt("int_neg"); print x x = readFloat("float"); print x x = readFloat("float_neg"); print x
When I run the python code, it reads these values correctly:
  ./tryMmapStruct.py 


testing... 6 -6 6.76999998093 6.78000020981

Posted by Kurt | Permalink

05.23.2005 08:27

Nanoblogger comment system

I have not tried this, but comments for nanoblogger

My Boring Ass Life - Silent Bob Speaks in his blog.

Posted by Kurt | Permalink

05.22.2005 19:26

In NOLA

I'm here for the AGU/SEG conference. Please come by my poster tomorrow afternoon and Leah Hogarth has hers tomorrow morning.

I am having trouble finding ANY Two Dog beer.

C++ Exceptions scare me still. I use exceptions in python, but never really tried in C++.

Cruise on over to google labs if you have not already. The only thing I would warn against is their desktop search tool and web accelarator. Large chances of google caching personal information.

RentACoder - maybe a way to earn cash.

MySQL tips

How Effective is Your Test Automation?

Using Performance Test Tools for High Volume Automated Testing

Posted by Kurt | Permalink

05.21.2005 21:57

Off to AGU New Orleans

Monday starts AGU/SEG.

Please stop by the Monday afternoon GP poster session and check out my magnetic susceptibility poster.

For you folks still using Matlab: Matlab contest winners

Posted by Kurt | Permalink

05.20.2005 21:58

New version of global paleomag database

The Global paleomagnetic database Bersion 4.6, February 2005 is out. I do not use this myself right now, but is cool to have available. Only bummer is that it is avaiable as an access database... that would mean that I would have to go use a Windows XP machine. Or is there something in the open source world that will read MS Access database files? There looks to be some stuff on freshmeat.net that can read MS Access databases. The one that seems to be the most likely for success is: mdbtools. The mailing list has a number of emails from this year and 0.6pre1 is from last summer. This looks to do the trick, but I have not tried it.

Royalty free stock photos

Posted by Kurt | Permalink

05.20.2005 13:56

Not My Workstation OS - Irix

In the past, I was a hard core IRIX/SGI fan. Those were the days when NASA put a $50K machine on my desk (Octane MXI - Dual 200MHz R10K). I was just exploring SGI's OpenInventor 2.x and the SGI user interface was way beyond Mac OS (was it 7 or 8 in 1996/7). There was the freeware CD which was so-so but worked, I build gcc to cross compile with 68K VxWorks, and we could use the new MipsPro compilers to do parallel processing for our stereo code. Life was good. Then I realized that I was using PC, Mac, VxWorks cage, Linux, SGI, and Sun all in one day. Yikes. Now I am a happy user of Mac OSX and Linux. The last stand for me with Irix is the 21 foot wide panoram screen in the viz center. Otherwise, Irix was a fantastic OS. It still has many advancements that are yet to be seen on other OSes. Someday I should rewrite button fly for any OS so then I can have flashbacks to my early days with Irix. I will never forget learning computer graphics at Stanford on those "super fast" R3K machines in the basement of Sweet Hall and the old computer science building.

My Workstation OS: Irix by Robert Mertling-Blake who apparently is a huge Irix fan. More power to him.

Posted by Kurt | Permalink

05.20.2005 09:15

Wave farm in Portugal

I thought there were already some wave farms going, like in Scotland, but this is pretty cool.

Portugal to get world's first commercial wave farm

And some news about the people who killed the Giant Sea Bass in La Jolla. Finally San Diego does something smart. Fishing in these areas hurts the local protected populations and hurts the tourist industry that this area counts on. Now for making the Children's Cove back into a protected seal area! The seals are a huge tourist attraction and there are plenty of other beaches for people to go to!
  FOR IMMEDIATE RELEASE:  May 19, 2005 
  Contact:  Maria Velasquez, Communications Director:  (619) 235-5725
  (pager & voicemail) mvelasquez@sandiego.gov 


CRIMINAL CHARGES FILED AGAINST SEA BASS POACHERS Complaint filed by San Diego City Attorney's Office

San Diego, CA: The Office of the San Diego City Attorney has filed criminal charges against three San Diego men involved in the April 24, 2005, killing of a protected Giant Sea Bass taken from the La Jolla Marine Conservation Area. Omid Adhami, 34 years old, has previously admitted to using a spear gun to kill the protected fish. Nima Hodaji, 26 years old, was diving with Adhami at the time the fish was killed. Navid Adibi, 22 years old, was operating the boat that took the two divers into the Conservation Area.

The Criminal Complaint filed in San Diego Superior Court alleges that all three men assisted in loading the 171 pound protected Giant Sea Bass onto the boat in the Marine Conservation Area off of La Jolla. Two counts of misdemeanor conspiracy have been filed against all three defendants who face up to one year in jail and a $10,000 fine on each conspiracy charge. According to Deputy City Attorney Kathyrn Lange of the City Attorney's Consumer and Environmental Protection Unit, Adhami faces a total four misdemeanor counts (three years in jail and $22,000 in fines), Hodaji and Adibi each face three misdemeanors counts (two and one-half years in jail and $21,000 in fines.) Arraignment of the three defendants will take place on June 2, 2005, in Department 1 of the San Diego Superior Court.

Giant Sea Bass are described as docile, curious, and slow moving fish. They are a protected species off of the California Coast. The Giant Sea Bass killed on April 24, 2005, was a reproductive female between 15 and 20 years old. The suspicious activity of the three men was first noticed by the City of San Diego Lifeguards located at the La Jolla Cove. The case is under investigation by the California Department of Fish and Game.


Apple battery recall:
  The recalled batteries include those with model numbers A1061,
  A1078, and A1079 and serial numbers that begin with HQ441 through
  HQ507 and 3X446 through 3X510


Veusz - yet another python plotting package.

We really need a Mac version of the python win32 video capture package. Working with the quicktime api directly is very frustrating.

Terminal from OSX 10.4 DVD - Nice. Now I need time to install 10.4.

PythonCallenge! - Also need more time for this.

Posted by Kurt | Permalink

05.20.2005 05:16

more on the butterflies

I have not seen these butterflies since my last post on there, but saw an article on the painted lady butterflies: Butterfly Migration Could Be Largest Known

Earthquake possibility in northern LA: Los Angeles 'Big Squeeze' Continues, Straining Earthquake Faults

A python unit conversion program that includes magnetic units: gonvert. I need to test the magnetics units. Here are some of the units:
Magnetic Field strength :
	ampere per meter
	oersted
Magnetic Flux :
	microweber
	milliweber
	unit pole (electro magnetic unit)
	weber
Magnetic Flux Density :
	gauss
	tesla
	weber per square meter

Posted by Kurt | Permalink

05.19.2005 08:09

New multilanguage SQLite interface API

SQLitepp "is a multilanguage object oriented wrapper to the sqlite library. It supports self-updatable records and a simple SQL-based query method." This was just announced on freshmeat.net

Python on the Playtation?


Posted by Kurt |
Permalink

05.18.2005 18:34

php

I did not know until just now that you can use php as a command line scripting language and it is available on on Mac OSX by default:
$ type -a php


php is /usr/bin/php

$ php -v

PHP 4.3.10 (cli) (built: Jan 23 2005 21:23:17) Copyright (c) 1997-2004 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

type -a php5

php5 is /sw/bin/php5 (this is fink)

$ php5 -v

PHP 5.0.4 (cli) (built: May 15 2005 12:55:54) Copyright (c) 1997-2004 The PHP Group Zend Engine v2.0.4-dev, Copyright (c) 1998-2004 Zend Technologies
To get you started, here is a very simple program to get started with: test.phps
#!/usr/bin/php
<?
$str="World";
print "Hello $str\n";
?>
Which you can run like this.
  ./chmod +x test.phps
  ./test.phps
And you will see something like this
./test.phps 


Hello World
Now, I really have no need to write php5 scripts, but it is nice to know that I can if I want to :)
Tom's Networking on How To Crack WEP - Part 2: Performing the Crack

Posted by Kurt | Permalink

05.18.2005 07:13

Computer Graphics World - May 2005

Today is May 18th - the 25th anniversary of the big Mt. St. Helens eruption. At 8:32 Sunday morning, May 18, 1980, Mount St. Helens erupted.

Computer Graphics World is one of those skinny little trade rags that say they cost but always end up giving you free subscriptions that never end. The May 2005 issue has some interesting things worth mentioning.

First, I have been thinking that there has to be someone doing a survey of new visualization and UI (User Interface) designs being used in computer games. There are like 12000 playstation games to date. That is a huge amount of engineering energy that had to get the look and feel right. How can we harness this for the marine and geo-sciences? So what is the very first text in this months issue? Exactly this topic. The "editor's note" is about Marc Prensky who wrote Digital Game-Based Learning. He goes through the top 10 nuggets. They did not do much for me, but the idea is right!

Then there is an add by 3Dconnexion for the SpacePilot. I have been thinking about a visualization workstation that combined a spacemouse for one hand and a dial for the other. I was thinking there are a lot of untried ideas for a dial:
  • could dial time in the model - just like my Sony VCR (remember VCR's right?)
  • spin along a flight path
  • dial in a brightness
  • how to navigate a color space?
  • spin an attached view, or...
  • add yet one more degree of freedom to camera motion with a spacemouse
  • and lots more
Damn... I never finished that opensource spacemouse driver that I tried to do last year. Anyone have a nice, free, and open driver? The 3dconnexion guys said they were only going to support USB on the mac and I have a serial spacemouse. Doh. Come on guys. Keyspan serial ports are cheap and show up as /dev/usb*.

"nobody-ever-got-for-buying-intel-world" - Those folks are missing out on some great stuff!

For NASA spacecraft, that would be: "nobody ever got fired for using VxWorks"

URL dmg attaching for Mac OSX:
  After reading the hdid man (via man hdid) page, I realized that we
  could mount .DMG images via HTTP under Terminal.app. And even
  cooler, the segmented images work too. Just reference the http-based
  DMG via this command: 
  % hdid http://server.company.com/Images/stuff.dmg


...

so the correct command is hdiutil attach URL. works fine for me.

Posted by Kurt | Permalink

05.18.2005 05:08

Fun with MS Windows

Arg. On our windows XP box we have been having yet more "issues." "ViewMgr" keeps popping up in zone alarm.

ViewMgr info

So why does each program need to have it's own updater that stays running on a system. That seems insane. If every program that one user had on a machine did the same thing, the machine would be clobbered just by the shear number of processes. Then there is another thing driving me crazy. A person was trying to use an GUI based ftp program to upload some stuff to a site. First off, they all want $30-$50 for an ftp program. You have got to be kidding. WinSCP is free and kicks all of their butts. If only the machine she were using allowed sftp... doh. Then, when she runs these ftp programs, norton says:
  Default Block Bla Trojan Horse 
  c:\WINNT\system32\ntvdm.exe
  Port 1042
Norton then gives you no further help or explanation. It looks like it is complaining about active ftp. Most users will opt to block this "trojan" and then have ftp not work. Is this the right conclusion? If not, PLEASE, someone set me straight!

Many tutorials: http://www.pixel2life.com/

Air travelers stripped bare with X-ray machine

New Collision Looks Imminent For B-15A Iceberg

http://www.projectorcentral.com/video_signals.htm - The Difference Between HDTV, EDTV, and SDTV

Posted by Kurt | Permalink

05.17.2005 13:52

Find cheap gas

http://www.ahding.com/cheapgas/ is down probably because of too much traffic, but this is what we all want. Where is that cheap gas? And it uses google maps too.

gasbuddy.com ... any good? Points to:

http://www.sandiegogasprices.com/ - none of these prices are anywhere near us :(

Posted by Kurt | Permalink

05.17.2005 11:08

more python, pysqlite 2.0

Wow... not yet, but I can't wait to dig into this soon: pysqlite 2.0.0 [freshmeat]

For me back in the old days, there is this: basic get-started guide for pysqlite 1.x

a pyblosxom setup

python imap email libs part 2

Posted by Kurt | Permalink

05.17.2005 10:07

python

An attempt at Constant types in Python

Work on a pyblosxom manual - pyblosxom - need time to play with this code!

AMD and Intel processors demistified at anandtech

Looks like rock positioning nicely controlling the location of wind generated ripples:


Posted by Kurt | Permalink

05.17.2005 08:15

SQL delete

This was my first chance to use the SQL DELETE command. Up until now, the only removal I needed to do was to delete the entire database as I regenerate it in about 30 sections from all of the flat file data tables. Now my problem is that I am on a deadline and crafting a way to compose a graph of remeasured data is a bit of a challenge (to me at least). I do not want to comment out the older values from the weights table that now how new measurements, but if I remove them from the table, I can later remove one function call from python to nuke that remove set of calls.

Here is one of the removes that I would like to do. For all of section 2 of core 3, I want to remove the first drying where the samples were just set out at room temp to dry (step=1). Since then, I dried the samples in the Norris and Jackson ovens at 50 °C for 2.75 days:

Update 2009-Jan-29: The new URL... http://vislab-ccom.unh.edu/~schwehr/Gaviota/bpsio-Aug04/measuring/

2005-May-16-drying photos

By the way, drying in the fume hood was only started the samples drying. Two days in the fume hood or the vacuum chambers only removed a small fraction of the water.

So how to make the query? Here are the schemas for the weights and the ams tables. I need the ams table since weights does not know about sections, cores, and other handy stuff:
  sqlite db ".schema weights"


CREATE TABLE weights (id INTEGER PRIMARY KEY, samplename VARCHAR(40), weight REAL, timeMeasured TIMESTAMP, status VARCHAR(20), operator VARCHAR(20), sticker VARCHAR(15), step INTEGER);

sqlite db ".schema ams"

CREATE TABLE ams (id INTEGER PRIMARY KEY, samplename VARCHAR(40), user VARCHAR(40), datemeasured TIMESTAMP, cruise VARCHAR(20), corenum INTEGER, coretype VARCHAR(1), corehalf VARCHAR(1), section INTEGER, sectionoffset REAL, depth REAL,counts INTEGER, sampleholder REAL ... and so forth
The final delete query:
  sqlite db "DELETE FROM weights WHERE weights.id IN (SELECT\
  weights.id FROM weights,ams WHERE weights.samplename=ams.samplename\
  AND corenum=3 AND section=2 AND weights.step=1);"
Before I deleted the samples, I can see lots of the ones I want to be rid of. There should only be
  sqlite db "select weights.id,weights.samplename,weights.step from\
  weights,ams where weights.samplename=ams.samplename AND corenum=3\
  and section=2 AND weights.step=1;" | tail -6


310|bp04-3gw-s2-132|1 311|bp04-3gw-s2-135|1 312|bp04-3gw-s2-138|1 313|bp04-3gw-s2-141|1 314|bp04-3gw-s2-144|1 315|bp04-3gw-s2-147|1
After the delete call here is what I see. All is well I hope!
  sqlite db "select weights.* FROM weights,ams WHERE weights.samplename=ams.samplename AND step=1 AND corenum=3 AND section=2"


Nothing

sqlite db "select weights.* FROM weights,ams WHERE \ weights.samplename=ams.samplename AND step=1 AND corenum=3" | head -3

316|bp04-3gw-s1-004|9.7838|2005-04-25 10:30:00|wet|schwehr|no-sticker|1 317|bp04-3gw-s1-007|9.8447|2005-04-25 10:30:00|wet|schwehr|no-sticker|1 318|bp04-3gw-s1-010|9.9916|2005-04-25 10:30:00|wet|schwehr|no-sticker|1

sqlite db "select weights.* FROM weights,ams WHERE \ weights.samplename=ams.samplename AND step=2 AND corenum=3" | head -3

561|bp04-3gw-s2-123|5.9082|2005-05-16 09:14:00|dry|schwehr|sticker|2 562|bp04-3gw-s2-126|5.8825|2005-05-16 09:14:00|dry|schwehr|sticker|2 563|bp04-3gw-s2-141|5.5927|2005-05-16 09:14:00|dry|schwehr|sticker|2
Now I need to account for the weight of the stickers.

Posted by Kurt | Permalink

05.17.2005 06:18

M. Chan on Mars Marbles



Yesterday, we had an interesting talk at SIO by Marjorie Chan entitled: "Analogs of Utah Marbles to Mars Blueberries." She makes a strong case for the spheroids on Mars being ground water derived concretions, but I am still not convinced and even if it was ground water, it may not have been very much. I still need to do some serious reading before I officially weigh in on this debate.

Wow. Someone from Moscow sucked down half a gig of stuff from my webserver yesterday. Hello to Moscow!

With spam the way it is today, if you actually won a contest, you would never know.

Google maps + tranceroute - give it a while to bring up the map. Especially today :)
UI testing resources

WASTE Network:
  WASTE (waste.sf.net) is software for setting up small, secure,
  private filesharing networks. WASTE provides chat, instant
  messaging, and file transfers between connected users, all of which
  is encrypted. The software is entirely peer to peer so no server is
  required. WASTE facilitates connections between NATed users with
  routing through addressable nodes. It works best for transferring
  files between small groups of people, like a group of friends. 

Posted by Kurt | Permalink

05.16.2005 19:21

PS3!!!

I am sure that if you care at all about the PS3, you have already scene this, but I'll put it here anyway. Alex and I spent a lot of time back in 1999/2000 talking about what we could do if we went in for a PS2 dev kit. I have never seen any of the products we thought up. Maybe I can get motivated for the PS3. What about a PS3 on the bridge of every major ship? It would no longer be just a game.

Anandtech on the PS3. Launch set for 2006.

It looks a little skimpy on RAM, but the unit should be a screamer. And with BlueRay disks, it will have a lot of space for data files. Plus a 2.5 inch HD will make the thing nice and flexible. Ah, 3 GigE port, 6 USB ports and WiFi should make integration a snap. Nice.

The real questions are what will be the price of the machine of the machine be, how much are dev kits, and how hard will it be to program for someone not in a game dev shop?

Posted by Kurt | Permalink

05.16.2005 14:49

Managing many computers

Here is a slashdot article where the comments might actually be useful in the future: Updating Free Software in the Enterprise?
  wallykeyster asks: "I'm an IT Director for a small private
  university in the U.S., and we are largely a Microsoft shop. We pay
  over $15,000 each year for our Campus Agreement so that we can
  upgrade the desktop OS to our version of choice, run Office, and
  have some Client Access Licenses. I would like to move to FOSS
  solutions, but I'm having trouble finding support for Enterprise
  management. For example, OpenOffice and Firefox (both of which I use
  personally) would be easy first steps, but IE is updated
  automatically via our SUS server (and settings pushed to clients via
  group policies) and Office updates will be included soon. How are
  other larger organizations (i.e. more than 200 desktops) dealing
  with software deployment and updates? Is anyone using Zen with
  Novell Desktop Linux?"
Maybe someday, I will actually look at CSS and see what it really is... Advanced CSS Layouts: Step by Step

Ask macslash: Bacula for mac and windows?
  ZEB17 writes "Anybody out there have any experience using Bacula to
  back up a Mac? How about with Tiger? We've been Retrospect users
  forever, but we find their upgrade and support policies a bit tough
  to swallow. Plus we've never been all that impressed with their
  software. So we started thinking about the open source project
  Bacula. It is supposed to work well on heterogenous networks with
  Linux, Windows, and Mac clients. But doing the usual searches
  reveals little about actual Mac usage. Is it so good that nobody
  ever posts anything? Or is it so bad that nobody even tries it?
  Thanks for any info."

Posted by Kurt | Permalink

05.16.2005 11:07

inside an G4 TiBook

TizGutz web page has an xray of the guts of a G4 Titanium laptop:



A chapter on the chapter online about the Mach system from the very good book 'Operating System Concepts'

How-To: BroadCatching using RSS + BitTorrent to automatically download TV shows - not for Macs, but you can adapt, right?

amail - anonymous email

Posted by Kurt | Permalink

05.16.2005 06:13

grid resampling in python

Extracting a low resolution grid from a high resolution grid

GreaseMonkey for Firefox. I still use Safari, but this sounds pretty cool but does it have serious security issues?

Coral Distribution Network - interesting network of web proxies. Have your page cached.

Posted by Kurt | Permalink

05.15.2005 13:20

python doctest

For some reason, I just decided that it is time to start adding doctest examples into my water analysis code. Then I discovered that I am not really understanding how this is supposed to work. I thought my array slicing function would be a good starting point. After a step back, I have a working example of doctest.
#!/usr/bin/env python


def subsetList (list, min, max): """ return a subregion of a list of (x,y) pairs based on the x range

>>> subsetList([(1,11),(2,22),(12,1212),(15,1515),(21,2121),(30,3030)], 10,20) [(12, 1212), (15, 1515)] "
"" newList=[] for item in list: if item[0]>=min and item[0]<=max: newList.append(item) return newList

def _test(): import doctest, foo doctest.testmod(foo)

if __name__ == '__main__': _test()
Which I then run by doing:
  chmod +x foo.py
  ./foo.py -v


Trying: subsetList([(1,11),(2,22),(12,1212),(15,1515),(21,2121),(30,3030)], 10,20) Expecting: [(12, 1212), (15, 1515)] ok 2 items had no tests: foo foo._test 1 items passed all tests: 1 tests in foo.subsetList 1 tests in 3 items. 1 passed and 0 failed. Test passed.
The key was to add "-v" after the command name to get more output as to what is going on through the tests.

PyStripchart looks very handy. Too bad it is not in fink.

Just noticed that there is python code for Artificial Intelligence: A Modern Approach by Peter Norvig. This is in addition to Lisp and Java. I have an older copy of this book on my shelf.

More free web hosting: http://pho2hosting.com/

Posted by Kurt | Permalink

05.15.2005 10:04

stumbler - viha

I tried to install Viha 0.0.1a and gave stumbler a try. I really just wanted to see how many wifi access points are in view and what channels they are using. No go:
  ./Stumbler
  IOConnectMapMemory (ipc/send) invalid destination port (10000003)
  Abort trap
And then I run it the top right menu bar goes blank and then comes back. Scary. Time to throw out the framework. Maybe I need to get and build 0.0.2 from source, but I am not going to spend the time to try that.

Instead, I got MacStumbler to work right off. Turns out that just about everyone in range is using channel 6. Not good! The only thing about macstumbler that I don't like is that it is actively interrogating all channels and devices. kismet/Viha does a passive grab of packets.

I never heard of Forensic Psychology until now:
  Forensic Psychology is the application of the science and profession
  of psychology to questions and issues relating to law and the legal
  system. The word "forensic" comes from the Latin word "forensis,"
  meaning "of the forum," where the law courts of ancient Rome were
  held. Today forensic refers to the application of scientific
  principles and practices to the adversary process where specially
  knowledgeable scientists play a role.
File Format List

Linux tips most of which are good for Mac OSX too.

Forumer - Free forums if you want to run an online discussion.

Posted by Kurt | Permalink

05.14.2005 14:07

Python Numeric exp not faster?

I thought the whole point about Numeric.exp() over the standard python Math.exp() was that it is faster. I figured I would do a quick check since I will be doing a lot of fitting to exponential functions. I got that Numeric is much slower. This is on a 1.5 GHz G4 Ti laptop with 1GB ram and 10.3.9 using python 2.4.1-1 from fink and numeric-py24 23.8-11. First the code. It maybe that there is some internal optimization that I am cancelling out by calling exp this way.
#!/usr/bin/env python
import math
import time
import Numeric


totalMath=0. totalNumeric=0.

for j in range(1,100): start = time.time() for i in range(1,100): x = math.exp(float(i)+.1) end = time.time() totalMath += end-start

start = time.time() for i in range(1,100): x = Numeric.exp(float(i)+.1) end = time.time() totalNumeric += end-start

print " totalMath = ",totalMath print " totalNumeric = ",totalNumeric
You just can not get any simpler than that. I do not think that python does any optimization that would just throw out the exp call since it is really a no-op. With C++, I would have to go look at the generated assembly to make sure the call actually happened. Now for the results:
./speed.py 
 totalMath    =  0.064
 totalNumeric =  0.192
Ouch :( max=700 print max print Numeric.exp(max) print math.exp(max)

max=800 print max print Numeric.exp(max) print math.exp(max) gives
  700
  1.01423205474e+304
  1.01423205474e+304
  800
  inf
  Traceback (most recent call last):
    File "./speed.py", line 17, in ?
      print math.exp(max)
  OverflowError: math range error

Posted by Kurt | Permalink

05.14.2005 12:05

ScientificPython fit and fit from gnuplot.py

I was wanting to use fit results from gnuplot in python code to calculate the residuals to an exponential. I was thinking of doing something like what I did with ams.py/k15_hext that uses popen2 to grad the results. That works, but I am thing popen2 is kind of ugly and it was slow. What does gnuplot.py use to talk to gnuplot?
  As the others have pointed out, you can call any gnuplot command from
  Gnuplot.py.  The other issue with "fit", however, is how do you get the
  results of the fit back into Python?  This is a problem because
  Gnuplot.py doesn't have access to gnuplot's standard output.
  
  Luckily this is easy--you write the parameters to a "parameters file"
  (which fortuitously is valid python code) then read them back into
  python.  This is untested but should approximately work:
  
    >>> g = Gnuplot.Gnuplot()
    >>> open('fit.par', 'w').write('m=0\nb=0\n')
    >>> g('fit m*x+b "data.in" via "fit.par"')
    >>> g('update "fit.par" "fit2.par"') # saves fitted parameters to fit2.par
    >>> parameters = {}
    >>> execfile('fit2.par', globals(), parameters)
    >>> print parameters
  {'b': 1.5015, 'm': 1e-30}
    >>> print parameters['m'], parameters['b']
  1e-30 1.5015
  
  In principle you could also parse "fit.log" to get the gory details
  about the fit (standard errors, correlation matrix, etc).
  
  Hope this helps,
  Michael
  
  -- Michael Haggerty mhagger at alum dott mit.edu
FiPy - FiPy: A Finite Volume PDE Solver Using Python

PyBLD - not that I really know what BLD is, but it does have a data fit capability.

GnuplotBiDir.py:
  Sometimes it is useful to get variables from a gnuplot session
  steered from python, e.g. to extract parameters from a
  fit. GnuplotBiDir.py is a test of bidirectional interfacing with
  gnuplot. For this you need a really recent gnuplot (from CVS) . The
  basic usage is 
    from GnuplotBiDir import Gnuplot
    gp=Gnuplot()
    gp("a=10")
    print "var=",gp.getvar("a")
  Remark: for an excellent python interface to gnuplot see gnuplot-py
  and the interactive add-ons to this from IPython. 
Numeric And Scientific packages [python.org]

MayaVI is in fink.
MayaVi is a free, easy to use scientific data visualizer. It is
written in Python and uses the amazing Visualization Toolkit (VTK) for
the graphics. It provides a GUI written using Tkinter.
ScientificPython does nonlinear least squares fits.

Module Scientific.Functions.LeastSquares

7.6.4 Scientific.Functions.LeastSquares. From BoA - The Bolometer Data Analysis Project. If every project had such documentation!
  >>> from Scientific.Functions.LeastSquares import *
  >>> from Numeric import exp
  >>> def f(param, t):
  ...     return param[0]*exp(-param[1]/t)
  ...
  >>> data = [(100, 4.999e-8),(200, 5.307e+2),
    (300, 1.289e+6),(400, 6.559e+7)]
  >>> print leastSquaresFit(f, (1e13,4700), data)
  ([8641551709749.7666, 4715.4677901570467], 1080.2526437958597)
I tried the above and it seems to work! I will probably be skipping gnuplot for fitting from now on. It is the same algorithm between the two. We will see if I take to this or not!

And another interesting mention: SciGraphica a scientific application for data analysis and technical graphics. It pretends to be a clone of the popular commercial (and expensive) application "Microcal Origin". It fully supplies plotting features for 2D, 3D and polar charts.

Wow. Too many plotting options!

Posted by Kurt | Permalink

05.13.2005 19:12

TOR

TOR = Tor: An anonymous Internet communication system. Uses "onion routers."

Is your port 9100 open? Look out. I wonder if we could make them run grid

:)

An Opportunity rover wheel that is stuck:


Posted by Kurt | Permalink

05.13.2005 06:31

Weight % water, Mail.app ssl, bash cheat sheet

Yesterday, I finally measured the weight of the "dry" samples. After talking to a few people, I realize that there are better ways to get this information. I should have taken samples straight from the oven at 50 degrees C and put them quickly into a vacuum desiccator. Instead, my samples were in atmosphere for up to 2 hours before being weighed. I found this document describing procedures from the Puget Sound project.
  Sedimentary Protocols [pdf]
Also, a higher temperature (>= 55° C) will get rid of some volatile organics.

Google dry weight procedure for marine sediments

I do not think the keychain stuff I did yesterday for Mail.app on Mac OSX worked. Maybe I need to logout/log back in? I need to dig in the keychain and make sure the cert is actually in there. Some relevant links: I tried again to add the UCSD pem file, but keychain says that it is already in the list. What gives? I still get a cert complaint from Mail.app.

And yet more Bash stuff: Bash Programming Cheat Sheet

Human-Computer Interaction(HCI)/Viz people that are on my list of people to lookup:
  • Tom Furness - uman Interface Technology Lab at UW.
  • Alex Pang at UCSC. Research interests: tensor visualization, scientific visualization, collaboration software, uncertainty visualization, virtual reality interfaces.
  • Terry Winograd

Posted by Kurt | Permalink

05.12.2005 09:29

OpenInventor syntax highlighting

I made an offhand post this morning to the coin discussion list and wow did it get responses! And that was just in an hour. I wanted to know if it would be possible to emit syntax highlight for open inventor code,

  Bernhard Breinbauer:


Maybe also not what you are looking for, but kwrite/kate from the KDE project have syntax highlighting for vrml files, and also a HTML export function. It's not the direct way, but you get coloured html output :-)
  Morton:


Not exactly what you're looking for, but in case it would be useful: Emacs comes with a VRML mode, which also works with Inventor files. As they usually do, the Emacs mode comes with syntax highlighting and indentation. Very useful.
  Chris Scharver


GNU enscript is another option. It can pretty-print the file and output to RTF, HTML, or PostScript as needed. It defaults to vrml syntax for .wrl files, but you can force it to use that mode for .iv files too. Very handy! I use it quite a bit on Linux, Mac OS X, and Windows via cygwin.

http://mega.ist.utl.pt/~ic-cg/laboratorio/vrmlTutorials/tutorial/source/tut7.html
I figured I might as well throw up an ugly version of my python xcore code for interested parties to look at. This code is UGLY:
  http://schwehr.org/xcore/xcore-py-0.1.tar.bz2


From the parental news wire service:

Mars rover struggles to escape sand dune

Easily import self-signed SSL certificates on Mac OSX. About time! I am tired of complaints from my mail server!
  With the new Safari in Tiger, it is easy to import self-signed SSL
  certificates you may come across. Just click Show More in the alert
  box, and then drag the certificate icon to a folder or your
  desktop. Then double-click the certificate, and Keychain Access will
  prompt to import it. Select the "X509Anchors" keychain from the
  Select box and click OK. Finally, enter your admin password to allow
  that keychain to be modified.


For example, my university has a self-signed SSL certificate for their IMAP server, and Mail.app constantly complains about this. So I pointed Safari at the IMAP SSL port eg: https://imap.ufl.edu:993/ to grab and import the certificate into the system keychain. Now when I start up Mail.app, it doesn't complain when connecting securely to my school's IMAP server

Posted by Kurt | Permalink

05.12.2005 08:13

OpenInventor LineSet

This is a quick exploration of the OpenInventor/Coin LineSet (man SoLineSet). I want to be able to simplify my usage of IndexedLineSet when I am not reusing vertices and I also want to be able to do reasonable effecient scatter plots so I will take a look at PointSet. I do not know if PointSet is in SGI's OpenInventor, but I am hoping that it is.

For LineSet, here is the example straight out of the Coin documentation:
#Inventor V2.1 ascii


Separator { Coordinate3 { point [ 0 0 0, 1 1 1, 2 1 1, 2 2 1, 2 2 2, 2 2 3, 2 3 2, 2 3 3, 3 3 3 ] } LineSet { numVertices [ 3, 4, 2 ] } }
The above example makes three disconnected lines. What if we have the simpler case of just one line for all the vertices that we specify? All you have to do is change the numVertices to be -1!
#Inventor V2.1 ascii


Separator { Coordinate3 { point [ 0 0 0, 1 1 1, 2 1 1, 2 2 1, 2 2 2, 2 2 3, 2 3 2, 2 3 3, 3 3 3 ] } LineSet { numVertices [ -1 ] } }
Now for a PointSet (man SoPointSet):
#Inventor V2.1 ascii


Separator { Material { diffuseColor [ 1 0 0, 0 1 0, 0 0 1, 1 1 0, 1 0 1, 1 1 1, 1 0.8 0.6, 0.6 0.8 1 ] } MaterialBinding { value PER_PART }

Coordinate3 { point [ -1 1 0, -1 -1 0, 1 -1 0, 1 1 0, 0 2 -1, -2 0 -1, 0 -2 -1, 2 0 -1 ] }

DrawStyle { pointSize 3 }

PointSet { } }
Why is it that we had a separate DotCloud application back in 1996-1997? Did Lew Hitchner all sorts of features that I never knew about (which is probably true) or was it a more fundamental problem with OpenInventor not having a PointSet back then?

Posted by Kurt | Permalink

05.12.2005 06:55

Another title about 'stuff'

pyparsing got a positive but terse review. More python parsing tools

POP and IMAP with python

Do not forget to download the latest firefox. 1.0.4 is out.

Dutch universities are going open. However, the actual website is all in dutch: DAREnet.

Kraftwerk

Posted by Kurt | Permalink

05.11.2005 18:49

Wireless woe

So I can't seem to get a reliable wireless connection to the basestation about 15 feet away. I think there are just too many base stations all trying to use the same freqeuencies. Is there a decent tool for Mac OSX that will tell me the real scoup on what it sees?

I am looking right now at how to do a funky self join in SQL. We will see if I can figure it out.... after about 15 minutes, here is what I have. I am guessing that there probably is a much better (read more efficient) way to rewrite this. First the table schema:

  .schema mag
  CREATE TABLE mag ( samplename VARCHAR(40), treatment REAL, csd REAL,
  intensity REAL, dec REAL, inc REAL, timeMeasured TIMESTAMP,  type
  VARCHAR(10), operator VARCHAR(20),  depth REAL,  cruise VARCHAR(20),
  corenum INTEGER, coretype VARCHAR(1), corehalf VARCHAR(1) ,  section
  INTEGER, sectionoffset REAL);
I want arm from the table and irm from the table for the same samplename. But I want the arm to always be from the same treatment of 100. This makes a table alias ARM and a table alias IRM table which I bring together:
  SELECT arm.samplename,arm.intensity,irm.intensity
   FROM mag arm 
   INNER JOIN mag irm 
   ON arm.samplename=irm.samplename 
   WHERE arm.type='arm'
    AND irm.type='irm' 
    AND arm.treatment=100
    AND arm.corenum=5 
   ORDER BY arm.depth;
Which gives results like this with the middle column being ARM 100 and the right column being IRM.
  ...
  bp04-5gw-s2-126|0.0000317600|0.0014030000
  bp04-5gw-s2-129|0.0000277200|0.0011360000
  bp04-5gw-s2-132|0.0000372000|0.0014940000
  bp04-5gw-s2-135|0.0000351100|0.0013400000
  bp04-5gw-s2-138|0.0000301600|0.0011740000
  bp04-5gw-s2-141|0.0000346500|0.0014260000
  bp04-5gw-s2-144|0.0000361300|0.0014160000

Posted by Kurt | Permalink

05.11.2005 16:29

mmap in python

Terry Fong and Marsokhod in the news: Overview of NASA's robots at BoingBoing. Thanks to ACD.

Lots of great data at the California Geological Survey: southern region quads. Thanks Roi! Check out San Diego on the right side: pdf

Wow. I did not know that python had easy access to mmap. This bodes well for my ideas on writing a generic low level driver for reading segy files quickly (maybe with writing too). I had been thinking that I needed some sort of low level C/C++ code to allow fast access into a file, but not if I can mmap in the file!

It is funny that this article is trying to get around a lack of solid IPC in MS Windows. I did not realize that windows had mmap. Now I need 64 bit mmap.

Communicate between processes using mmap

More on matplotlib which I think is more flexible than Gnuplot.

Posted by Kurt | Permalink

05.11.2005 06:25

Python/emacs

An emacs mode for spyce. So I really don't understand the lisp code, but it sounds interesting and it really is a short snippet. This follows after I was thinking yesterday about emacs modes.

Hidding mp3's in gifs

Python challenge solutions, part 1

Flashkit - If you use flash, this is supposed to have lots of cool stuff. I would like to play with flash someday. It looks very cool. Need more time/day!

Posted by Kurt | Permalink

05.10.2005 19:36

A9.com Block View

Saw an interesting article in the June Discover: "Come Fly With Me." It talks about Google's Keyhole flyover software that uses keyhole satellite data and the A9.com (Amazon) Block View system where you can go up and down a block.

Here is Frankie Johnnie & Luigi Too in Mountain View where Roy, Ruwan and I have met up many times. I think there should be a wiki for posting funny pictures from Block View. Find anyone you know driving by in the picture?

More on bash by Ken Steube at the SDSC.

I really want a nicer OpenInventor emacs mode. Maybe someday I will read this mode tutorial and write a real mode. I did write a mode for Arc Macro Language (AML), but I have long since lost the code. Maybe it is still hiding on the USGS WR sierra machine.

Posted by Kurt | Permalink

05.10.2005 15:40

Schneier on Cryptography

Bruce Schneier on Cryptography at Security Focus.

YellowIcon - another free icon site.

Yellowstone Rated High for Eruption Threat

Free Wendy's frosty this weekend

Tomsnetworking on how to crack wep - Remember, just because you are running wep someone may be listening to your traffic.

Posted by Kurt | Permalink

05.10.2005 09:15

Python data visualization

The linux gazette has an interesting article: Python for scientific use. Part I: Data Visualization

WordPress 1.5.1 released. This to try after I graduate.

Scary: MIT Profs, Colleagues Propose Plan For Nuclear Energy. What is the true solution for long term global energy needs?

Previously Unknown Fault Provides New Insights On Himalayan Mountain Building

Surf Not Up For Palaeozoic Creatures - New Model Reveals Ancient Sea Was A Giant Lake

Google maps weblogs

10 Big Myths about copyright explained. I should read this.

More bash craziness. Part 4. If I was still doing my whole database work from bash, I would devour this article. As it is, I'll stick to python. But Ben writes very well and bash is my favorit shell (I never want to go back to csh,tcsh,or the most evil or evil's - DOS)

More on knoppix... what to use when you can't deal with MS windows any more.

Work on Opportunity to get it out of a dune

Posted by Kurt | Permalink

05.09.2005 14:20

Joann Stock on the Gulf of California

Got to hear Joann Stock talk about the Gulf of California today. Great talk!

unrealid

Spacial Metaphore - debates about finder and all

VideoEditing

Working today on drying out my samples to get weight percent water. Not going as fast as I hoped. I now have two vacuum balls full of samples. I put some fresh dryright in one of them to see if that makes a difference.

Photos of drying.

Posted by Kurt | Permalink

05.09.2005 05:22

ALIA-KM0506 Samoa Expedition

Check out the web page for the Samoa expedition:

ALIA

Someone on the fink dev mailing list posted a picture that was being served up by gopher. Yikes. Flashbacks to 1993.

gopher://sdf.lonestar.org/00/users/newmanbe/Random_Pictures_I_Have_Made/fin=kmap.png

Posted by Kurt | Permalink

05.08.2005 21:00

Gantt charts and other notes

GanttProject

Using Ant rather than GNU Make. Will I ever give up make files?

Did some more IRM's today on the Santa Barbara cores. Finished up core 6 and 1. Started those two sets of cubes drying in the fume hood. Need to get them setup in vacuum drier to make sure.

Two notes from Becca about xraying... news from the Gulf of Mexico


Just read your blog and we got some software from the xray guy on the cruise - Barry. He uses some free software called ImageJ

http://rsb.info.nih.gov/ij/download.html

it is medical software but is very useful for viewing xrays in tif format and Barry wrote a macro using the pixel intensity (~density) to calculate out the cracks and voids in the core to get the true length. I have not really used it yet but he swears by it so definetly worth a look!
  Barry brought a ct and a really really high res xray machine that he
  built on the boat - it can xray cores that are stil under pressure
  in an autoclave! and then if that is not enough he can also do "time
  lapse" measurements of the core degassing as the pressure is
  released slowly.  All sorts of techno gradgetry is going on!

Posted by Kurt | Permalink

05.07.2005 13:40

4D visualization/capture

The Hungry Mind: Prof. Avideh Zakhor on being a science nerd in Iran, why Larry Summers made her mad, and what the heck a 4D model is. Interesting content is at the bottom. She is trying to capture both the spacial component along with the temporal changes. Here are the key links. Some of her work is here:

UC Berkeley Video and Image Processing Lab

Next Generation, 4-D Distributed Modeling and Visualization - vismuri

An older Slashdot article: Automatic 3D Reconstruction of Scenes about the instant scene modeler which looks like they are using like the gray point system, no?

I need to look through the available open source 3D software again to see what the possibilities are as I do not think the Ames Stereo Pipeline will be released any time soon.

Quicktime Pro 6 Key with most of 7 - Use a backup of Quicktime Pro 6 on a machine with 7. The old app will use the new codecs.

15 things you can do with RSS. Don't forget to scroll down to the comments too.

Learning morse code with a binary tree. It has been about 17 years since I took the morse code test. Back then I only passed because I guessed what the scentence was from the few letters that I got right. It would be fun to be really good at morse code. I would love to be able to practice by entering text into my computers in morse via a keyer. That would be a fun and very simple microcontroller project.

Back to processing the new core photo...

Posted by Kurt | Permalink

05.07.2005 11:13

Preventing owners from data

The same goes with cars and ships. I think it is not right to disallow the owner of a car, ship, or plane from the diagnostic data for the vehicle or certain parts.

Engine Monitor Data Change Upsets Owners

Cracking the realestate code

Posted by Kurt | Permalink

05.07.2005 10:27

Reshooting BPSIO-04 gravity cores

Yesterday Liz helped me reshoot the BPSIO-04 cores. We first scraped the surface of the cores to remove wire drag marks and the older oxidized surface. The hope is that this would reveal many more features in the core. I really should have done this when I split the cores, so I will definitely remember that for next time I collect cores!

To do the scraping, we removed the plastic marker plugs and used a piece of metal lightly dragged across the surface. We tried both with and without the fluorescent lights and think that without them produces a better image.

Photos of the process were taken with the paleomag digital camera - an Olympus Camedia Z3000 Zoom 3.3 MPixel model that is now a couple years old.


Posted by Kurt | Permalink

05.06.2005 08:49

mosaicing xrays

I have been thinking about the issues with digital xrays of cores. What we have done works pretty well, but we can always do better. There are several goals that I have in mind for processing xrays.
  • Make nice mosaics for publications
  • Make mosaics that best show off all of the features in the core
  • Be able to make quantative statements about xray density
I haven't really figured out any of these issues to any satisfaction, but I have lots of ideas.

This morning, I ran into a document on the web about compositing xray images from paintings using a system called VIPS:

Working With Digital X-ray Images: Some practical image manipulation using VIPS/ip 7." by Joseph Padfield. Some issues are the same, some are not. It would be great to be able to do a good job with whole round or half round cores by correcting for both thickness and xray emmission issues(e.g. where on the plate the pixel comes from).

Posted by Kurt | Permalink

05.06.2005 07:51

Network toolkits and depth level in Open Inventor

Terry Fong based me a CMU tech report yesterday that is pretty cool. Jay Gowdy compares network transport packages for robotics.

A Qualitative Comparison of Interprocess Communications Toolkits for Robotics

It is missing a some of the things that I consider with network toolkits and it is a bit dated at 2000, but definitely a great read.

On another note, I just added a draggable depth line to my xcore displays. I was getting annoyed with trying to figure out what depth an object is at and I did not want to go write a ray pick call back. This is based on the floating ocean for my Eel River / Humboldt 2002 project.
#Inventor V2.1 ascii
# Small IV file to give a depth reference line so that I stop
# spending so much time guessing depths.

Separator {
    Separator {
	Transform {
	    rotation	0 -1 0  1.57
	}
	DEF +2 Translate1Dragger {}
    }
    Separator {
	Translation { translation -20 0 5 }
	Scale { scaleFactor .3 .3 .3 }
	RotationXYZ { axis X angle 1.57 }
	Text3 { string "Depth in cm" = 
		Calculator {
		    C	0 0 0 =
		    USE +2
		    . translation
		    expression	"oa = -C[0]"
		}
		. oa
	    }
    }
    Translation { translation 5 -.2 0 }
    Transform {
	translation	0 0 0 =
	Calculator {
	    C	0 0 0 =
	    USE +2
	    . translation
	    expression	"oA = vec3f(0,0,C[0])"
		}
		. oA


} Coordinate3 { point [ 0 0 0, 300 0 0, ] } # Coord3 IndexedLineSet { coordIndex [ 0, 1, -1, ]} # IndexedLineSet }

Posted by Kurt | Permalink

05.06.2005 05:16

Mars Polar Lander found?


Posted by Kurt | Permalink

05.04.2005 13:47

python stuff

tclpython - Calling python code from within Tcl.

Contact sheet of images using PIL. Maybe this will be better than what I was doing with JWZ's code that would not work do to a broken image magick on OSX.

Search Resumes for the Mars Polar Lander (MPL)

Posted by Kurt | Permalink

05.03.2005 02:49

Dulles airport

It's 2:40 AM PST in the Dulles airport with 2 hours until my next flight, so it is time to catch up a on few links sitting in my laptop's dock.

Lake Tahoe makes the top of NSF's news.

Z800 3D Visor - Yet another head mounted display. No head tracking?

Freephotoshop.com - Brushes, etc.

Geologists' Find Is Evolutionary
  Geologists believe that Earth experienced at least three global ice
  ages from 800 to 580 million years ago (mya) during the
  Neoproterozoic Era, at the end of the Precambrian Age. In most
  places, evidence of the glaciations has disappeared, but traces
  remain in several areas with "cap carbonates," thin layers of
  limestone containing distinctive ratios of carbon isotopes. "When
  the ice melts, the oceans rapidly precipitate carbonate, which
  appears in the sedimentary rock," explains one of the geologists,
  Daniel Condon, a postdoctoral fellow in geology/geochemistry. "Cap
  carbonates are very unusual and restricted to glacial periods in the
  Neoproterozoic."
GCC 4.0 review

Hackers guide to QT

How well do you know python - Say what? Maybe it is that I am reading this in the middle of the night.

Posted by Kurt | Permalink

05.01.2005 06:59

why updatedb has been filling my disk

I have been dealing with daily find/sort issues stemming from the fink locate/updatedb system. Here are some posts about this.

>>>> When I run updatedb from findutils in fink, I get this error:
>
>>> Does /usr/bin/find have the same problem?
>
>> It doesn't say it does.
>
> Sorry to bother everyone, but found a fix. I added fsdec to the list 
> of fstypes to prune, and it worked fine (/dev/fd/3 couldn't be 
stat'd)
> Maybe we can still track down why it says the hard link count is 
> incorrect though; if it doesn't have that error there is a very nice 
> speed boost because it doesn't stat each file then.
>
Hey folks,


I'd like to add that this is a *critical* bug, and it occurs on 10.3, too -- maybe the new findutils version introduce the problem?

Today and yesterday my HD run full. The problem is that apparently, for updatedb /dev/fd/3 is an alias for /. So when updatedb scans my FS, it gets into an infinite recursion loop. In my case it created about 15 GB of temp files (indexing the "infinitely many files" on my HD), before it run out of HD space.

Running out of HD space is rather dangerous, if you notice it too late it can cause data loss (10.3 really tries to warn you about this, but if your machine is running while you are absent, as in my case, you can be too late)...


Hula Girl hot sauces

Posted by Kurt | Permalink