Tuesday, December 29, 2009
nasty spaces in paths
But I admit I don't get the $1=$1.
tcsh, etc:
foreach file (`find , -name "* *"`)
mv "$file" "`echo $file | awk 'BEGIN {OFS="-"} $1=$1'`"
end
bash, etc:
find . -name "* *"|while read file
do
mv "$file" "`echo "$file"| awk 'BEGIN {OFS="-"} $1=$1'`"
done
Wednesday, December 23, 2009
Wednesday, December 16, 2009
My Current Favorite TED Talk
A very interesting talk. Significantly outside the box. Interesting approach to the following:
- Evolution of morality.
- How can morality be usefully parsed?
- What is morality good for?
Monday, September 14, 2009
sed to the rescue (python old-style classes are new again)
So a lot of my code has: class English1A:
instead of: class English1A (object):
I finally got tired of having a mix of old and new classes and used sed to fix it:
sed -i .bak 's/^\(class .*[^()]\):/\1 (object):/' *py
Monday, August 17, 2009
Firefox 3.5 = Blank tooltips everywhere
--UPDATE--
To Fix: uninstall GoogleToobar; reinstall new version of GoogleToolbar (For me-5.0.20090813W)
Tuesday, August 11, 2009
Wednesday, July 22, 2009
Dork Haus - Case Mods? Furniture Mods?
Is it a case mod, or are you just making your otherwise useful furniture much more dorky? We have 2 Dork Haus examples. The motivation is that in the Man Cave there are a lot of tools and servers, and the servers have media files on them that we dorks may want to enjoy. The Dork Haus Man Cave is not one of these TV sissy man caves with comfy chairs, carpet and football paraphernalia. We put the "cave" back in "man cave" where it belongs (tools and servers, babe).
Great, but I would rather relax in the den and the bedroom, hence the need for computers in those rooms. Computers are ugly (yes, even your mac... especially where the finish is worn off from you petting it) or if not ugly at least inappropriately styled for most decor. I don't want to make the computer more of an issue (e.g. these case mods). I want the computer, the blinking, the noise (Oh God, the noise!) to go away.
With the release of the new ATOM based motherboards I decided to revisit the idea of making a very small unobtrusive computer. I had tried earlier with a VIA board that turned out to be too anemic to playback video but still got its case hot, scary hot.
The Den of Dork:
The small chest of drawers the upper left of the tv-holder-thing has a computer in it. Unfortunately, the TV had no available inputs on the back, so I had to use the s-video input on the side of the TV. The bottom two drawers have a little INTEL motherboard and fashioned connections for video, power, audio, USB and RJ45 on the back of the cabinet. The "remote control" is a small wireless keyboard which is stored in the drawer above the computer and the top drawer has the TV and DVD-player remotes in it.
She does make a little noise as the motherboard has a fan and the hard drive is a real hard drive (that spins), and so far I haven't made a nice switch, so you have to open the drawer to reach a switch scavenged from an old tower-style PC. This is probably for the best, as you want to leave the drawer cracked when it's on for ventilation. Pictures of the back.
It's running windows XP, which is probably about as much as that processor can handle. One thing that is nice about running Windows it that it works with Netflix "Watch Instantly." Also the networking is brain-dead simple with SAMBA running downstairs on FreeBSD 7.2. I have to say that Rhapsody runs very poorly on this setup; but other players (including WMP) work great and it is more than up to the job of video playback with any codec over the ~800x600 s-video line. The last picture is Internet Explorer at my Google Reader page.
The Master Dork Bedroom:
Observe: an armoire. She is an old and relatively nice armoire, and has been doing a fine job holing up a TV while containing clothes within. This particular item used to be my brother Neal's, and it ended up moving with us to Salt Lake City at the near insistance of my mom. It is about 5 feet tall and it is made mostly of thick boards (not plywood). We like it a lot - thanks mom!
Obviously the computer would need to go in or on the armoire. There is a hatch on the upper right that opens up to reveal a mirror and a generous area to hold your razor or something. I bolted a mother board to the back of that space with 1" standoffs. This board only requires 12VDC for power which simplifies things considerably. Also this set up has no fan and I'm using a solid state hard drive. If the speakers are off, it makes no noise that I can hear.
Julie, the She-Dork, recommend that rather than make lots of little holes and connections I make a single simple hole and pass the cables right through. This was much easier and required much less abusing of the furniture. I made a single rectangular hole about 1"X1.5", and that easily accommodates the 120VAC cord for the power brick, the CAT5e, HDMI, USB and stereo audio cables. One more hole allowed a sweet little power switch on the back of the armoire.
Thursday, July 16, 2009
Phifer the kitten
Tuesday, June 30, 2009
tee functionality with python subprocess PIPEs
One example application is dumping large databases with storage engine that can't give you a time of last modification, e.g. mysql, using innodb. Then I want the chksum of the dump, and if that is the same as the dump before, I'll unlink the older one and symlink it to the new dump. I also want to check the dump stream to make sure it has finished properly. And for good measure lets compress the stream on the fly. SURE, you could do all this after the dump completes but then you have to wait for a lot of disk I/O that the pipes avoid.
If you are like me, you'll want to use python. Note that if you take the stdout from one pipe and read it with more that one (n) other process, each process will only get a fraction of the data, ~1/n. So use a buffer to store the data and write it to each process that needs it.
__version__ = [int(x) for x in "$Revision: 1.2 $".split()[1].split('.')]
__author__ = "Kael Fischer <kael.fischer@gmail.com>"
.
.
.
syslog.syslog(syslog.LOG_ERR,"INFO: Starting mysqldump of %s on %s" % (db,host))
try:
# make output files
outFile = file(outfileName,'w')
digFile = file(digestName,'w')
# make pipes
dumper= subprocess.Popen(["mysqldump","--opt","--skip-dump-date",
"--single-transaction","--quick",db],
stdout=subprocess.PIPE)
grepper = subprocess.Popen(["grep", '-q','^-- Dump completed$'],
stdin=subprocess.PIPE,stdout=sys.stdout)
digester = subprocess.Popen(["md5", "-q"],stdin=subprocess.PIPE,stdout=digFile)
bziper = subprocess.Popen(["bzip2"],
stdin=subprocess.PIPE,stdout=outFile)
while dumper.poll() == None:
# use os.read NOT file.read
# file.read blocks.
# copy output from mysqldump to a buffer
buf=os.read(dumper.stdout.fileno(),5000)
# write buffer contents to 3 different processes
grepper.stdin.write(buf)
digester.stdin.write(buf)
bziper.stdin.write(buf)
# after dumper finishes,
# explicitly shut down input streams
# to other jobs
grepper.stdin.close()
digester.stdin.close()
bziper.stdin.close()
while grepper.poll() == None:
# wait if needed (shouldn't be)
time.sleep(1)
if grepper.returncode != 0:
raise RuntimeError("End of dump not found: grep returned - %s"%
grepper.returncode)
except BaseException, e:
# dump's no good, KeyboardInterrupt, whatever
syslog.syslog(syslog.LOG_ERR,"ERROR: %s : %s" % (type(e),e))
syslog.syslog(syslog.LOG_ERR,"ERROR: mysqldump of %s on %s failed" % (db,host))
# unsuccessful
# put files back where they were
# and exit
# exercise for reader
syslog.syslog(syslog.LOG_ERR,"INFO: mysqldump of %s on %s finished" % (db,host))
Monday, June 29, 2009
python subprocess based parallel processing
This is a jiffy to run a shell command on a number of hosts.
#!/usr/local/bin/python -u
#
# runAllOver.py
# Run a shell command on several machines using ssh
#
__version__ = tuple([int(x) for x in
'$Revision: 1.2 $'.split()[1].split('.')])
__author__ = "Kael Fischer"
import sys
import time
import optparse
from subprocess import Popen, PIPE
HOSTS = ["nfs1","nfs2","compute1","compute2","db1" ]
def main(sysargs):
oneLineUsage = "Usage: %prog [options] '<remote command>'"
op = optparse.OptionParser(
oneLineUsage,
version="%prog " + '.'.join([str(x) for x in __version__]))
(opts,args) = op.parse_args(sysargs)
try:
if len(args) == 0:
raise RuntimeError, "No remote command specified."
except Exception, eData:
print >> sys.stderr, ("\nUsage Error: %s\n" %eData.message)
print >> sys.stderr, op.format_help()
return 1
cmd = ' '.join(args)
print cmd
# make one running pipe object per host
pipes = [remotePipe(h,cmd) for h in HOSTS]
# report the results in turn
for i,p in enumerate(pipes):
print HOSTS[i] +':'
while p.poll() == None:
time.sleep(0.5)
print p.stdout.read()
return(0) # we did it!
def remotePipe(host,cmd,block=False):
p=Popen("ssh %s '%s'" %(host, cmd),shell=True,stdout=PIPE)
if block:
while p.poll() == None:
time.sleep(1)
return p
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
What's cool about that? Well all the processes are running on the hosts simultaneously and that makes it go fast.
Isn't that insecure? Could be, depending on the context. For ways to secure that kind of thing more, read this article by Brian Hatch: http://www.hackinglinuxexposed.com/articles/20021211.html. It was the basis for the intermachine communication in the PHABRIX and most especially prun was built using his authprogs as a starting concept (with greater flexibility and extra security layers added).
Wednesday, June 24, 2009
Monday, June 22, 2009
Thursday, June 18, 2009
Tuesday, June 16, 2009
Sunday, May 24, 2009
Monday, May 18, 2009
Hawk Alert - Dork Haus
Saturday, May 16, 2009
Zombies?
This is not a death threat or anything like that. I just like zombies.
zombama.jpg (JPEG Image, 250x389 pixels)
Monday, May 11, 2009
Friday, May 8, 2009
Nate you are a major geek!
Horray! The Second Derivative of the Unemployment Rate Improved!
Friday, May 1, 2009
4.N.13
__iter__ goes through the digits - allowing gui code to build digit boxes easily.
e.g.:
>>> print maths_4_N_13.LongDivFormatted(15670,3)
5,223
-------- R=1
3 | 15,670
>>>
from types import IntType, LongType
THOUSANDS_SEP = ','
DECIMAL_SEP = ''
class TeachingWholeNumber(object):
"""An integer whole number (>= 0) where each digit is addressable
list style. the 0th element of the 'digits' attribute
is the ones place, the 1st element is the
tens place and so on.
"""
def __init__(self,n=0):
"""New object with default value of 0
"""
if n == 0:
self.digits=[0]
else:
n=int(n)
self.digits=[]
while n > 0:
self.digits.append(n%10)
n=n/10
def __int__(self):
"""Returns the corresponding integer value
"""
retValue = 0
for i,v in enumerate(self.digits):
retValue += (10**i)*v
return retValue
#
# listy methods
#
def __len__(self):
"""Returns the length of the digits list
"""
return len(self.digits)
def __getitem__(self,i):
if type(i) not in (IntType,LongType):
return TypeError, "Index must be an integer type"
else:
return self.digits[i]
def __setitem__(self,i,v):
if type(i) not in (IntType,LongType):
return TypeError, "Index must be an integer type"
if type(v) != IntType or v > 9 or v < 0:
return ValueError, "value, v, must be an integer between 0 and 9"
if i < len(self):
self.digits[i]=v
elif v > 0:
# appending 0 past the last non-zero is a null op
while i> len(self):
self.digits.append(0)
self.digits.append(v)
#
# Expected methods for number-like type
#
def __lt__(self, other):
return int(self) < int(other)
def __le__(self, other):
return int(self) <= int(other)
def __eq__(self, other):
return int(self) == int(other)
def __ne__(self, other):
return int(self) != int(other)
def __gt__(self, other):
return int(self) > int(other)
def __ge__(self, other):
return int(self) >= int(other)
def __add__(self, other):
return self.__class__(int(self)+int(other))
def __sub__(self, other):
return self.__class__(int(self)-int(other))
def __mul__(self, other):
return self.__class__(int(self)*int(other))
def __floordiv__(self, other):
return self.__class__(int(self)/int(other))
def __mod__(self, other):
return self.__class__(int(self)%int(other))
def __divmod__(self, other):
return tuple([self.__class__(x)
for x in divmod(int(self),int(other))])
def __pow__(self, other):
return
def __lshift__(self, other):
return NotImplemented
def __rshift__(self, other):
return NotImplemented
def __and__(self, other):
return NotImplemented
def __xor__(self, other):
return NotImplemented
def __or__(self, other):
return NotImplemented
def __div__(self, other):
return self.__class__(int(self)/int(other))
def __truediv__(self, other):
return NotImplemented
def __neg__(self):
return NotImplemented
def __pos__(self):
return NotImplemented
def __abs__(self):
return abs(int(self))
def __invert__(self):
return NotImplemented
def __complex__(self):
return NotImplemented
def __long__(self):
return long(int(self))
def __float__(self):
return float(int(self))
def __repr__(self):
return str(int(self))
def __str__(self):
"""pretty string
"""
dgtList=[]
for i,v in enumerate(self):
if i>0 and (i-1)%3 == 2:
dgtList.append(THOUSANDS_SEP)
dgtList.append(str(v))
dgtList.reverse()
return ''.join(dgtList)
def rshift(self,places=1):
"""right shift (in base 10) disgarding 'places' least significant digits
Arguments:
- `self`:
- `places`:
"""
# is this better implememted by overriding >> ?
while places > 0:
self.digits.pop(0)
places -= 1
def lshift(self,places=1):
"""left shift (in base 10) disgarding 'places' least significant digits.
Arguments:
- `self`:
- `places`:
"""
# is this better implememted by overriding << ?
while places > 0:
self.digits.insert(0,0)
places -= 1
def randomTWN(low,high):
"""factory fcn to return a random TeachingWholeNumber,
between low and high inclusive.
"""
if high < 0 or low < 0:
raise ValueError, "high and low must be > 0"
i = random.randint(low,high)
return TeachingWholeNumber(i)
class Division (object):
"""Division Container
"""
def __init__ (self,dividend,divisor):
"""New one!
"""
self.dividend = dividend
self.divisor = divisor
self.quotent, self.remainder = divmod(dividend,divisor)
class LongDivFormatted(Division):
"""Knows how to print itself
"""
leftMargin = 1
rightMargin = 2
midMargin = 1
vertbar='|'
horizbar='-'
def __init__ (self,dividend,divisor,useTWN=True):
"""New one!
"""
if useTWN:
dividend=TeachingWholeNumber(dividend)
divisor=TeachingWholeNumber(divisor)
Division.__init__(self,dividend,divisor)
def __str__ (self):
"""ASCII representation.
"""
mainWidth = (self.leftMargin+
len(str(self.divisor))+
(self.midMargin *2)+
len(self.vertbar)+
len(str(self.dividend)))
remainderWidth = ((self.rightMargin) +
len(str(self.remainder))+
len('R='))
totalWidth = mainWidth+remainderWidth
retLines=[]
retLines.append(('%'+str(mainWidth)+'s')%(self.quotent))
retLines.append((' '*(self.leftMargin +
len(str(self.divisor))+
self.midMargin)) +
(self.horizbar*(len(self.vertbar)+
self.midMargin+
len(str(self.dividend)))) +
('%'+str(remainderWidth)+'s')%('R='+str(self.remainder)))
retLines.append((' ' * self.leftMargin) +
str(self.divisor) +
(' ' * self.midMargin) +
self.vertbar +
(' ' * self.midMargin) +
str(self.dividend))
return '\n'.join(retLines)
Wednesday, April 29, 2009
Tuesday, April 28, 2009
Humongous Python | O'Reilly Media
Humongous Python | O'Reilly Media
Monday, April 27, 2009
School of Medicine Seminars & Events - via biochemistry department
Wednesday, April 15, 2009
Monday, April 6, 2009
The Guantánamo Testimonials Project — The Center for the Study of Human Rights in the Americas (CSHRA)
Saturday, March 28, 2009
Syntax Highlighting Code in Webpages
If you are using Blogger (or somewhere else where you can't upload files). You can hot link the the scripts and css at http://alexgorbatchev.com/pub/sh/ which is up 99% of the time.
kdbom.tryDBconnect
It automatically checks localhost,3306 for the database after checking at the specified sites.... good if clients are not on the same private subnet the the server is on.
def tryDBconnect(db=None,serversPorts=None,user=None,
reuseDB=None,getTables=True,tryLocalPort=3306,
fatal=False,verboseFailure=False):
"""Try to connect to a database and return the db instance. One or more server/port
combinations can be tried. If no connection is successful a Warning is issued,
unless fatal is true in which case KdbomDatabaseError is raised.
db = databse name (string)
server = servers hostname (string)
port = server's listening port (int)
After all given (server,port) combinations are tried one or more localhost
connections over port tryLocalPort will be attempted unless that is set
to None. tryLocalPort can be an integre type or a sequence of integers.
serversPorts should be a list of (server,port) tuples, or a single 2-tuple.
serversPorts can also be a single string to make replacement of db()
calls more straight forward (defaultPort is then used for the port).
Ir reuseDB is specified, this is just a call to the db class constructor.
"""
connected = False
database = None
if reuseDB != None:
database = kdbom.db(db=db,reuseDBconnection=reuseDB,getTables=getTables)
connected = True
else:
if type (serversPorts) in StringTypes:
serversPorts = [(serversPorts,defaultPort)]
if type(serversPorts) != ListType:
if type(serversPorts) == NoneType:
serversPorts = []
elif (len(serversPorts) == 2
and type(serversPorts[0]) in StringTypes
and type(serversPorts[1]) in (IntType,LongType)):
serversPorts = [serversPorts]
elif type(serversPorts) == TupleType:
serversPorts = [serversPorts]
else:
raise ArgumentError, "serversPorts should be a list of tuples not: %s" % serversPorts
try:
for p in tryLocalPort:
serversPorts.append(('localhost',p))
except TypeError:
serversPorts.append(('localhost',tryLocalPort))
for host,port in serversPorts:
try:
print host,port
database = kdbom.db(db=db,host=host,port=port,user=user,getTables=getTables)
connected = True
except:
pass
if not connected:
if fatal:
if not verboseFailure:
raise KdbomDatabaseError, "%s DB not loaded" % db
else:
warnings.warn("%s DB not loaded" % db)
return database
Thursday, March 26, 2009
Google + Python + LLVM
Update:
More from PyPy's author.
Weather Station Near Dork Haus
Weather Station History : KUTSALTL17
Or at work it's like this -
Tuesday, March 24, 2009
GPL Harmful (the heresy of ESR)
dotCOMmie's Blog
Monday, March 23, 2009
Old Snooping Methods are New Again
Researchers find ways to sniff keystrokes from thin air | ITworld
Sunday, March 22, 2009
Thursday, March 19, 2009
What people really think about God
Here is a record of how people feel about Pastafarians: Hate Mail (and concerned criticism) archive - Church of the Flying Spaghetti Monster
Censorwatch: $11,000-A-Day Fine For Linking To A Blacklisted Site, Wikileaks Blacklisted - Gizmodo Australia
wikileaks
White House Declares Copyright Treaty State Secret | WebProNews
Tuesday, March 17, 2009
PEP 289 -- Generator Expressions
kewler than list comprehension: generator expressions (!)
PEP 289 -- Generator Expressions
Compare command output on 2 hosts
#!/bin/tcsh
set tmp1 = `mktemp -t computeDiff.csh.c1.XXXXXX`
set tmp2 = `mktemp -t computeDiff.csh.c2.XXXXXX`
onintr exit
foreach cmd ( $argv )
ssh kf-compute1 "$cmd" >! $tmp1
ssh kf-compute2 "$cmd" >! $tmp2
echo "########### $cmd ################"
echo "### diff compute1 compute2"
diff $tmp1 $tmp2
end
exit:
rm $tmp1* >&/dev/null
rm $tmp2* >&/dev/null
# end script
Q: What Software Is Installed On Only One Node?
A:
% computeDiff.csh pkg_info
########### pkg_info ################
### diff compute1 compute2
22a23
> compat6x-amd64-6.4.604000.200810 A convenience package to install the compat6x libraries
24d24
< cvsup-without-gui-16.1h_4 File distribution system optimized for CVS (non-GUI version
34d33
< fastest_cvsup-0.2.9_5 Finds fastest CVSup server
62a62
> iperf-2.0.4 A tool to measure maximum TCP and UDP bandwidth
150d149
< sge-6.2.b Sun Grid Engine, a batch queueing system
Thursday, March 12, 2009
finaly factored out PickleableObject
class PickleableObject(object):
"""Generic pickleable class
"""
def save(self,filename,clobber=False):
"""given an object, e.g. a taxon dictionary,
pickle it into the specified filename
Set clobber = True to allow overwriting of the
output file"""
import cPickle
f = safeOFW(filename,clobber=clobber,append=False)
cPickle.dump(self,f)
f.close()
@classmethod
def restore(cls,filename):
"""given a filename, return an object containing the
pickled data in that file"""
import cPickle
f = open(filename,'rb')
obj = cPickle.load(f)
return obj
I've been meaning to move this from a class that Peter wrote it in to originally at my request. It's quite nice to have it globally available.
Wednesday, March 11, 2009
Words that nerds use that I hate
tarball - "tar file" (the fact that it is gzipped, will be obvious to any that care)
architected - the word you are looking for is "designed" n.b. actual architects do not use this word to describe their activities.
sequel - Do you mean "S"-"Q"-"L"? (./ers you know the 'sequel' form is M$ spawn, right?)