Tuesday, December 29, 2009

nasty spaces in paths

This works pretty well:

But I admit I don't get the $1=$1.

tcsh, etc:
foreach file (`find , -name "* *"`)
mv "$file" "`echo $file | awk 'BEGIN {OFS="-"} $1=$1'`"
end

bash, etc:
find . -name "* *"|while read file
do
mv "$file" "`echo "$file"| awk 'BEGIN {OFS="-"} $1=$1'`"
done

Wednesday, December 23, 2009

Eureka!


The Splayd!

Can the the power structure really withstand the Spife?

Wednesday, December 16, 2009

My Current Favorite TED Talk

Jonathan Haidt on the moral roots of liberals and conservatives. (or: Why Everyone Wants To Go To Applebee's; Just No One You Know.)

A very interesting talk. Significantly outside the box. Interesting approach to the following:

  • Evolution of morality.
  • How can morality be usefully parsed?
  • What is morality good for?

Monday, September 14, 2009

sed to the rescue (python old-style classes are new again)

Python became my language of choice when new-style classes were still really new and not yet supported everywhere. Python seemed to work just fine and it was many years before I started using new-style classes.

So a lot of my code has: class English1A:

instead of: class English1A (object):

I finally got tired of having a mix of old and new classes and used sed to fix it:
sed -i .bak 's/^\(class .*[^()]\):/\1 (object):/' *py

Monday, August 17, 2009

Firefox 3.5 = Blank tooltips everywhere

After upgrading to Firefox 3.5 I have empty tooltips for what looks like every DIV. If you turn off browser.chrome.toolbar_tips in about:config it gets rid of them. It may get rid of _all_ tooltips, so watch out for that.

--UPDATE--
To Fix: uninstall GoogleToobar; reinstall new version of GoogleToolbar (For me-5.0.20090813W)

Wednesday, July 22, 2009

Dork Haus - Case Mods? Furniture Mods?

When you put a computer inside a piece of furniture, what is that?

Is it a case mod, or are you just making your otherwise useful furniture much more dorky? We have 2 Dork Haus examples. The motivation is that in the Man Cave there are a lot of tools and servers, and the servers have media files on them that we dorks may want to enjoy. The Dork Haus Man Cave is not one of these TV sissy man caves with comfy chairs, carpet and football paraphernalia. We put the "cave" back in "man cave" where it belongs (tools and servers, babe).

Great, but I would rather relax in the den and the bedroom, hence the need for computers in those rooms. Computers are ugly (yes, even your mac... especially where the finish is worn off from you petting it) or if not ugly at least inappropriately styled for most decor. I don't want to make the computer more of an issue (e.g. these case mods). I want the computer, the blinking, the noise (Oh God, the noise!) to go away.

With the release of the new ATOM based motherboards I decided to revisit the idea of making a very small unobtrusive computer. I had tried earlier with a VIA board that turned out to be too anemic to playback video but still got its case hot, scary hot.

The Den of Dork:
The small chest of drawers the upper left of the tv-holder-thing has a computer in it. Unfortunately, the TV had no available inputs on the back, so I had to use the s-video input on the side of the TV. The bottom two drawers have a little INTEL motherboard and fashioned connections for video, power, audio, USB and RJ45 on the back of the cabinet. The "remote control" is a small wireless keyboard which is stored in the drawer above the computer and the top drawer has the TV and DVD-player remotes in it.







She does make a little noise as the motherboard has a fan and the hard drive is a real hard drive (that spins), and so far I haven't made a nice switch, so you have to open the drawer to reach a switch scavenged from an old tower-style PC. This is probably for the best, as you want to leave the drawer cracked when it's on for ventilation. Pictures of the back.







It's running windows XP, which is probably about as much as that processor can handle. One thing that is nice about running Windows it that it works with Netflix "Watch Instantly." Also the networking is brain-dead simple with SAMBA running downstairs on FreeBSD 7.2. I have to say that Rhapsody runs very poorly on this setup; but other players (including WMP) work great and it is more than up to the job of video playback with any codec over the ~800x600 s-video line. The last picture is Internet Explorer at my Google Reader page.





The Master Dork Bedroom:

Observe: an armoire. She is an old and relatively nice armoire, and has been doing a fine job holing up a TV while containing clothes within. This particular item used to be my brother Neal's, and it ended up moving with us to Salt Lake City at the near insistance of my mom. It is about 5 feet tall and it is made mostly of thick boards (not plywood). We like it a lot - thanks mom!










Obviously the computer would need to go in or on the armoire. There is a hatch on the upper right that opens up to reveal a mirror and a generous area to hold your razor or something. I bolted a mother board to the back of that space with 1" standoffs. This board only requires 12VDC for power which simplifies things considerably. Also this set up has no fan and I'm using a solid state hard drive. If the speakers are off, it makes no noise that I can hear.













Julie, the She-Dork, recommend that rather than make lots of little holes and connections I make a single simple hole and pass the cables right through. This was much easier and required much less abusing of the furniture. I made a single rectangular hole about 1"X1.5", and that easily accommodates the 120VAC cord for the power brick, the CAT5e, HDMI, USB and stereo audio cables. One more hole allowed a sweet little power switch on the back of the armoire.


Tuesday, June 30, 2009

tee functionality with python subprocess PIPEs

Sometimes we may want the output of a process to go to more then one place. In a shell, we would probably use tee(1).

One example application is dumping large databases with storage engine that can't give you a time of last modification, e.g. mysql, using innodb. Then I want the chksum of the dump, and if that is the same as the dump before, I'll unlink the older one and symlink it to the new dump. I also want to check the dump stream to make sure it has finished properly. And for good measure lets compress the stream on the fly. SURE, you could do all this after the dump completes but then you have to wait for a lot of disk I/O that the pipes avoid.

If you are like me, you'll want to use python. Note that if you take the stdout from one pipe and read it with more that one (n) other process, each process will only get a fraction of the data, ~1/n. So use a buffer to store the data and write it to each process that needs it.


__version__ = [int(x) for x in "$Revision: 1.2 $".split()[1].split('.')]
__author__ = "Kael Fischer <kael.fischer@gmail.com>"

.
.
.


syslog.syslog(syslog.LOG_ERR,"INFO: Starting mysqldump of %s on %s" % (db,host))
try:
# make output files
outFile = file(outfileName,'w')
digFile = file(digestName,'w')

# make pipes
dumper= subprocess.Popen(["mysqldump","--opt","--skip-dump-date",
"--single-transaction","--quick",db],
stdout=subprocess.PIPE)
grepper = subprocess.Popen(["grep", '-q','^-- Dump completed$'],
stdin=subprocess.PIPE,stdout=sys.stdout)
digester = subprocess.Popen(["md5", "-q"],stdin=subprocess.PIPE,stdout=digFile)
bziper = subprocess.Popen(["bzip2"],
stdin=subprocess.PIPE,stdout=outFile)

while dumper.poll() == None:
# use os.read NOT file.read
# file.read blocks.
# copy output from mysqldump to a buffer
buf=os.read(dumper.stdout.fileno(),5000)
# write buffer contents to 3 different processes
grepper.stdin.write(buf)
digester.stdin.write(buf)
bziper.stdin.write(buf)

# after dumper finishes,
# explicitly shut down input streams
# to other jobs
grepper.stdin.close()
digester.stdin.close()
bziper.stdin.close()

while grepper.poll() == None:
# wait if needed (shouldn't be)
time.sleep(1)

if grepper.returncode != 0:
raise RuntimeError("End of dump not found: grep returned - %s"%
grepper.returncode)

except BaseException, e:
# dump's no good, KeyboardInterrupt, whatever
syslog.syslog(syslog.LOG_ERR,"ERROR: %s : %s" % (type(e),e))
syslog.syslog(syslog.LOG_ERR,"ERROR: mysqldump of %s on %s failed" % (db,host))

# unsuccessful
# put files back where they were
# and exit

# exercise for reader


syslog.syslog(syslog.LOG_ERR,"INFO: mysqldump of %s on %s finished" % (db,host))

Monday, June 29, 2009

python subprocess based parallel processing

The python threading module is cool and when combined with rpyc and the Sun Grid Engine you can get a lot done really fast on a cluster. I will blog that later, but using one of the tricks I use with rpyc with the newish 'subprocess' module in the python standard library, multi-process based parallel processing seems simpler then ever now.

This is a jiffy to run a shell command on a number of hosts.



#!/usr/local/bin/python -u
#
# runAllOver.py
# Run a shell command on several machines using ssh
#
__version__ = tuple([int(x) for x in
'$Revision: 1.2 $'.split()[1].split('.')])
__author__ = "Kael Fischer"

import sys
import time
import optparse
from subprocess import Popen, PIPE


HOSTS = ["nfs1","nfs2","compute1","compute2","db1" ]

def main(sysargs):

oneLineUsage = "Usage: %prog [options] '<remote command>'"

op = optparse.OptionParser(
oneLineUsage,
version="%prog " + '.'.join([str(x) for x in __version__]))

(opts,args) = op.parse_args(sysargs)


try:
if len(args) == 0:
raise RuntimeError, "No remote command specified."
except Exception, eData:
print >> sys.stderr, ("\nUsage Error: %s\n" %eData.message)
print >> sys.stderr, op.format_help()
return 1

cmd = ' '.join(args)
print cmd

# make one running pipe object per host
pipes = [remotePipe(h,cmd) for h in HOSTS]

# report the results in turn
for i,p in enumerate(pipes):
print HOSTS[i] +':'
while p.poll() == None:
time.sleep(0.5)
print p.stdout.read()

return(0) # we did it!

def remotePipe(host,cmd,block=False):
p=Popen("ssh %s '%s'" %(host, cmd),shell=True,stdout=PIPE)
if block:
while p.poll() == None:
time.sleep(1)
return p

if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))


What's cool about that? Well all the processes are running on the hosts simultaneously and that makes it go fast.

Isn't that insecure? Could be, depending on the context. For ways to secure that kind of thing more, read this article by Brian Hatch: http://www.hackinglinuxexposed.com/articles/20021211.html. It was the basis for the intermachine communication in the PHABRIX and most especially prun was built using his authprogs as a starting concept (with greater flexibility and extra security layers added).

Monday, May 18, 2009

Hawk Alert - Dork Haus

I think I saw a Sharp-shinned Hawk at Dork Haus Sunday morning, but it might have been a Cooper's Hawk.

Saturday, May 16, 2009

Friday, May 1, 2009

4.N.13

Apropos of the Sugar 4th grade math activity that David and I are working on, I wrote an integer like class that prints nicely and that allow each digit to be addressed as if it were a list.

__iter__ goes through the digits - allowing gui code to build digit boxes easily.

e.g.:

>>> print maths_4_N_13.LongDivFormatted(15670,3)
5,223
-------- R=1
3 | 15,670
>>>




from types import IntType, LongType

THOUSANDS_SEP = ','
DECIMAL_SEP = ''

class TeachingWholeNumber(object):
"""An integer whole number (>= 0) where each digit is addressable
list style. the 0th element of the 'digits' attribute
is the ones place, the 1st element is the
tens place and so on.
"""

def __init__(self,n=0):
"""New object with default value of 0
"""
if n == 0:
self.digits=[0]
else:
n=int(n)
self.digits=[]
while n > 0:
self.digits.append(n%10)
n=n/10

def __int__(self):
"""Returns the corresponding integer value
"""
retValue = 0
for i,v in enumerate(self.digits):
retValue += (10**i)*v
return retValue


#
# listy methods
#
def __len__(self):
"""Returns the length of the digits list
"""
return len(self.digits)

def __getitem__(self,i):
if type(i) not in (IntType,LongType):
return TypeError, "Index must be an integer type"
else:
return self.digits[i]

def __setitem__(self,i,v):
if type(i) not in (IntType,LongType):
return TypeError, "Index must be an integer type"
if type(v) != IntType or v > 9 or v < 0:
return ValueError, "value, v, must be an integer between 0 and 9"
if i < len(self):
self.digits[i]=v
elif v > 0:
# appending 0 past the last non-zero is a null op
while i> len(self):
self.digits.append(0)
self.digits.append(v)

#
# Expected methods for number-like type
#
def __lt__(self, other):
return int(self) < int(other)

def __le__(self, other):
return int(self) <= int(other)

def __eq__(self, other):
return int(self) == int(other)

def __ne__(self, other):
return int(self) != int(other)

def __gt__(self, other):
return int(self) > int(other)

def __ge__(self, other):
return int(self) >= int(other)

def __add__(self, other):
return self.__class__(int(self)+int(other))

def __sub__(self, other):
return self.__class__(int(self)-int(other))

def __mul__(self, other):
return self.__class__(int(self)*int(other))

def __floordiv__(self, other):
return self.__class__(int(self)/int(other))

def __mod__(self, other):
return self.__class__(int(self)%int(other))

def __divmod__(self, other):
return tuple([self.__class__(x)
for x in divmod(int(self),int(other))])

def __pow__(self, other):
return

def __lshift__(self, other):
return NotImplemented

def __rshift__(self, other):
return NotImplemented

def __and__(self, other):
return NotImplemented

def __xor__(self, other):
return NotImplemented

def __or__(self, other):
return NotImplemented

def __div__(self, other):
return self.__class__(int(self)/int(other))

def __truediv__(self, other):
return NotImplemented

def __neg__(self):
return NotImplemented

def __pos__(self):
return NotImplemented

def __abs__(self):
return abs(int(self))

def __invert__(self):
return NotImplemented

def __complex__(self):
return NotImplemented

def __long__(self):
return long(int(self))

def __float__(self):
return float(int(self))

def __repr__(self):
return str(int(self))

def __str__(self):
"""pretty string
"""

dgtList=[]
for i,v in enumerate(self):
if i>0 and (i-1)%3 == 2:
dgtList.append(THOUSANDS_SEP)
dgtList.append(str(v))
dgtList.reverse()
return ''.join(dgtList)

def rshift(self,places=1):
"""right shift (in base 10) disgarding 'places' least significant digits

Arguments:
- `self`:
- `places`:
"""
# is this better implememted by overriding >> ?

while places > 0:
self.digits.pop(0)
places -= 1

def lshift(self,places=1):
"""left shift (in base 10) disgarding 'places' least significant digits.
Arguments:
- `self`:
- `places`:
"""
# is this better implememted by overriding << ?
while places > 0:
self.digits.insert(0,0)
places -= 1

def randomTWN(low,high):
"""factory fcn to return a random TeachingWholeNumber,
between low and high inclusive.
"""

if high < 0 or low < 0:
raise ValueError, "high and low must be > 0"

i = random.randint(low,high)
return TeachingWholeNumber(i)


class Division (object):
"""Division Container
"""

def __init__ (self,dividend,divisor):
"""New one!
"""
self.dividend = dividend
self.divisor = divisor
self.quotent, self.remainder = divmod(dividend,divisor)


class LongDivFormatted(Division):
"""Knows how to print itself
"""
leftMargin = 1
rightMargin = 2
midMargin = 1
vertbar='|'
horizbar='-'

def __init__ (self,dividend,divisor,useTWN=True):
"""New one!
"""
if useTWN:
dividend=TeachingWholeNumber(dividend)
divisor=TeachingWholeNumber(divisor)
Division.__init__(self,dividend,divisor)


def __str__ (self):
"""ASCII representation.
"""
mainWidth = (self.leftMargin+
len(str(self.divisor))+
(self.midMargin *2)+
len(self.vertbar)+
len(str(self.dividend)))

remainderWidth = ((self.rightMargin) +
len(str(self.remainder))+
len('R='))

totalWidth = mainWidth+remainderWidth

retLines=[]
retLines.append(('%'+str(mainWidth)+'s')%(self.quotent))
retLines.append((' '*(self.leftMargin +
len(str(self.divisor))+
self.midMargin)) +
(self.horizbar*(len(self.vertbar)+
self.midMargin+
len(str(self.dividend)))) +
('%'+str(remainderWidth)+'s')%('R='+str(self.remainder)))
retLines.append((' ' * self.leftMargin) +
str(self.divisor) +
(' ' * self.midMargin) +
self.vertbar +
(' ' * self.midMargin) +
str(self.dividend))

return '\n'.join(retLines)

Tuesday, April 28, 2009

Humongous Python | O'Reilly Media

We recently had a pygame presentation at the Utah-PUG and it got me thinking about these Putt-Putt games my daughter was playing about the time I was first learning python. The company that made putt-putt used python for those games. They made and distributed a GPL'ed debugger to help with their C++/python projects.

Humongous Python | O'Reilly Media

coin slot detector

The essential coin slot detector.

Monday, April 6, 2009

The Guantánamo Testimonials Project — The Center for the Study of Human Rights in the Americas (CSHRA)

Tired of listening to blowhards (pundits, presidents, etc.) who talk about whether or not the American Government tortures people? If so, you may be interested in the actual testimonies of people who were there: The Guantánamo Testimonials Project — The Center for the Study of Human Rights in the Americas (CSHRA) Click through for actual knowledge.

Saturday, March 28, 2009

Syntax Highlighting Code in Webpages

There is a nice bit of Javascript code here: http://alexgorbatchev.com/wiki/SyntaxHighlighter This package allows uses css to display pretty source code on a webpage. It supports a lot of languages, but is missing a lot of common ones too.

If you are using Blogger (or somewhere else where you can't upload files). You can hot link the the scripts and css at http://alexgorbatchev.com/pub/sh/ which is up 99% of the time.

kdbom.tryDBconnect

I put a factory function for kdbom.db instances in kdbom.kdbom (previously I had it in viroinfo).

It automatically checks localhost,3306 for the database after checking at the specified sites.... good if clients are not on the same private subnet the the server is on.



def tryDBconnect(db=None,serversPorts=None,user=None,
reuseDB=None,getTables=True,tryLocalPort=3306,
fatal=False,verboseFailure=False):
"""Try to connect to a database and return the db instance. One or more server/port
combinations can be tried. If no connection is successful a Warning is issued,
unless fatal is true in which case KdbomDatabaseError is raised.

db = databse name (string)
server = servers hostname (string)
port = server's listening port (int)

After all given (server,port) combinations are tried one or more localhost
connections over port tryLocalPort will be attempted unless that is set
to None. tryLocalPort can be an integre type or a sequence of integers.

serversPorts should be a list of (server,port) tuples, or a single 2-tuple.
serversPorts can also be a single string to make replacement of db()
calls more straight forward (defaultPort is then used for the port).

Ir reuseDB is specified, this is just a call to the db class constructor.
"""

connected = False
database = None

if reuseDB != None:
database = kdbom.db(db=db,reuseDBconnection=reuseDB,getTables=getTables)
connected = True
else:
if type (serversPorts) in StringTypes:
serversPorts = [(serversPorts,defaultPort)]
if type(serversPorts) != ListType:
if type(serversPorts) == NoneType:
serversPorts = []
elif (len(serversPorts) == 2
and type(serversPorts[0]) in StringTypes
and type(serversPorts[1]) in (IntType,LongType)):
serversPorts = [serversPorts]
elif type(serversPorts) == TupleType:
serversPorts = [serversPorts]
else:
raise ArgumentError, "serversPorts should be a list of tuples not: %s" % serversPorts
try:
for p in tryLocalPort:
serversPorts.append(('localhost',p))
except TypeError:
serversPorts.append(('localhost',tryLocalPort))

for host,port in serversPorts:
try:
print host,port
database = kdbom.db(db=db,host=host,port=port,user=user,getTables=getTables)
connected = True
except:
pass

if not connected:
if fatal:
if not verboseFailure:
raise KdbomDatabaseError, "%s DB not loaded" % db

else:
warnings.warn("%s DB not loaded" % db)
return database

Thursday, March 26, 2009

Google + Python + LLVM

There has been talk of targeting python to LLVM before (PyPy). Google is jumping in with the Unladen Swallow Project, hoping for performance multiples in the 3-5x range.

Update:
More from PyPy's author.

Weather Station Near Dork Haus

I wonder what the weather at my house is like right now?

Weather Station History : KUTSALTL17


Or at work it's like this -

Tuesday, March 24, 2009

Monday, March 23, 2009

Old Snooping Methods are New Again

I guess this has been in the works for a while, but now it's out in the open. Remember: just because you are paranoid, that doesn't mean they aren't after you.

Researchers find ways to sniff keystrokes from thin air | ITworld

Google HaX0r

l33t hax0r google

600673.COM

Thursday, March 19, 2009

What people really think about God

It is interesting to see the hate people have for people who have the audacity to believe something they do not. Sunnite <=> Shi'ite; Christian <=> Pastafarian.

Here is a record of how people feel about Pastafarians: Hate Mail (and concerned criticism) archive - Church of the Flying Spaghetti Monster

FSM Bank Card - Church of the Flying Spaghetti Monster

What I want for my birthday!

FSM Bank Card - Church of the Flying Spaghetti Monster

Ample evidence shows Flying Spaghetti Monster is creator | News-Leader.com | Springfield News-Leader

REPENT!

Ample evidence shows Flying Spaghetti Monster is creator | News-Leader.com | Springfield News-Leader

Censorwatch: $11,000-A-Day Fine For Linking To A Blacklisted Site, Wikileaks Blacklisted - Gizmodo Australia

Censorwatch: $11,000-A-Day Fine For Linking To A Blacklisted Site, Wikileaks Blacklisted - Gizmodo Australia: "Censorwatch: $11,000-A-Day Fine For Linking To A Blacklisted Site, Wikileaks Blacklisted"

wikileaks

White House Declares Copyright Treaty State Secret | WebProNews

Obama not changing things.

White House Declares Copyright Treaty State Secret | WebProNews

Tuesday, March 17, 2009

PEP 289 -- Generator Expressions

Everyone that learned python from me, probably doesn't know these exist. So take note:

kewler than list comprehension: generator expressions (!)

PEP 289 -- Generator Expressions

Compare command output on 2 hosts

This works nicely on my 2 compute nodes.


#!/bin/tcsh
set tmp1 = `mktemp -t computeDiff.csh.c1.XXXXXX`
set tmp2 = `mktemp -t computeDiff.csh.c2.XXXXXX`
onintr exit
foreach cmd ( $argv )
ssh kf-compute1 "$cmd" >! $tmp1
ssh kf-compute2 "$cmd" >! $tmp2
echo "########### $cmd ################"
echo "### diff compute1 compute2"
diff $tmp1 $tmp2
end

exit:
rm $tmp1* >&/dev/null
rm $tmp2* >&/dev/null
# end script


Q: What Software Is Installed On Only One Node?
A:

% computeDiff.csh pkg_info
########### pkg_info ################
### diff compute1 compute2
22a23
> compat6x-amd64-6.4.604000.200810 A convenience package to install the compat6x libraries
24d24
< cvsup-without-gui-16.1h_4 File distribution system optimized for CVS (non-GUI version
34d33
< fastest_cvsup-0.2.9_5 Finds fastest CVSup server
62a62
> iperf-2.0.4 A tool to measure maximum TCP and UDP bandwidth
150d149
< sge-6.2.b Sun Grid Engine, a batch queueing system

Thursday, March 12, 2009

Kaelbot Central Moving Up In Rankings

I have my first follower!

finaly factored out PickleableObject

uses my nonclobbering file opener: safeOFW

class PickleableObject(object):
"""Generic pickleable class
"""
def save(self,filename,clobber=False):
"""given an object, e.g. a taxon dictionary,
pickle it into the specified filename

Set clobber = True to allow overwriting of the
output file"""
import cPickle
f = safeOFW(filename,clobber=clobber,append=False)
cPickle.dump(self,f)
f.close()

@classmethod
def restore(cls,filename):
"""given a filename, return an object containing the
pickled data in that file"""
import cPickle
f = open(filename,'rb')
obj = cPickle.load(f)
return obj



I've been meaning to move this from a class that Peter wrote it in to originally at my request. It's quite nice to have it globally available.

Wednesday, March 11, 2009

Words that nerds use that I hate

distro - "distribution" please
tarball - "tar file" (the fact that it is gzipped, will be obvious to any that care)
architected - the word you are looking for is "designed" n.b. actual architects do not use this word to describe their activities.
sequel - Do you mean "S"-"Q"-"L"? (./ers you know the 'sequel' form is M$ spawn, right?)

HealthMap | Global disease alert map

This should go on the lab web page HealthMap | Global disease alert map