damians personal blog

the other day i wanted to speed up the annoying task of repetitive dowloads from a website. i thought, since machines are built for completing repetitive tasks fast, my computer should be able to download some files on it's own.

after launching wget with the corresponding parameters i noticed that wget would fail because of the design of the website: when you arrived at the index page, there is a pull down menu in a (hidden) form, which will let you choose an entry. every entry will redirect you to the file which it had as value:

<form name="jump"><br />
<select name="menu" onChange="location=document.jump.menu.options[document.jump.menu.selectedIndex].value;" value="GO"><br />
<option value="first.html">first</option><br />
<option value="second.html">second</option>

the file you are redirected to had the interesting contents i wanted to download automatically - but wget will not recognize the value field in the select element as a local link, and thus will not follow it.

so i would have to prepare a html file which is comprehensive also for wget. i copied the options-part of the source code into my favourite text editor and replaced the tags <option value="first.html">first</option> analogous this schema <a href="http://www.thewebsite.com/first.html">first</a> .

i saved the modified code to ~/Sites/mylinkslist.html and invoked wget with the following arguments:

this worked perfectly fine and wget downloaded all the files i wanted automatically.

wget arguments shortly explained:
-nc, --no-clobber do not download file twice
-r, --recursive follow all the links on a webpage
-l, --level=NUMBER how many link-levels to follow
-H, --span-hosts go to foreign hosts when recursive
some more useful wget arguments:
-A, --accept=LIST accepted file extensions, comma-separated
-np, --no-parent do not ascend to parent directory in recursive mode
-c, --continue resume a partially-downloaded file

w¡nd0w$ user can find a win-ified version of wget here: http://gnuwin32.sourceforge.net/packages/wget.htm.
those who own a real computer (i.e. a mac) use fink or apt-get to install wget on their system.

        27.Jan.2005, 18:57 :: programming :: Permalink :: Comments (0)

to make this video widely spread, i will give a mirror for here:
macmini.mp4

        20.Jan.2005, 16:09 :: apple :: Permalink :: Comments (0)

oder eher: seislertüütschì spröchwörtar.

da ich seit nun etwas mehr als einem halben jahr 8 monate mit einer senslerin zusammen bin war, und auch sonst einige fribourger in meinem freundeskreis hab, habe ich beschlossen, so die besten sensler zitate aufzuschreiben, die mir zu ohren gekommen sind.

• "we dä länga wär, was är tumma ìsch, nei chentenär chnewliga us om tachchännù triicha."
  wenn der so lang (gross) wäre, wie er dumm ist, könnte er kniend aus der dachrinne trinken.
  
  
• "a maa oni buch ìsch we da hömmu oni stärna."
  ein mann ohne bauch ist wie der himmel ohne sternen.
  
  
• "wetti no epper ässe oder sölis grad diräkt de hùnä gäh?"
  will noch jemand davon essen, oder soll ich es direkt den hunden zum frass geben? (anspielung auf die kochkunst)
  
  
• "säg' mer nid mi name, säg' mer schwi du moora!"
  (er zu ihr) sag mir nicht meinen namen, sag mir schwein du sau!
  
  
• "a motta isch an härdchnödr met chrùtt."
  eine "motta" ist ein erdklumpen mit gras.
  
  
• "i chodera dier a rügg' dass du meensch du heegisch as grüens rucksäckli an!"
  ich spucke dir so sehr an den rücken, dass du meinst, du hättest einen grünen rucksack an!
  
  
• "du büsch a zeligi' titta."
  du bist ein wehleidiges tröpfchen.(vulgär)
  
  
• "'geng sövu' het's müsli gseet, wo's iz meer bislet het."
  'besser als gar nichts' sagte das mäuschen, derweil es ins meer pinkelte.

diese sammlung wird sich hoffentlich laufend vergrössern, dh das post wird immer wieder aktualisiert.

        20.Jan.2005, 0:13 :: unsorted :: Permalink :: Comments (0)

as it turned out, the configuration of my crontab i showed in the last post doesn't work correctly. it did shut down mlnet, but it didn't relaunch it correctly. also i wanted to launch mlnet from a tcsh and increase file descriptor count as described in the forum on macdk.com, but nothing really worked. so i came nearly back to the initial configuration, with the difference that now i relaunch it each 2 hours.

excerpt from my /private/etc/crontab

the script /Volumes/MAX120/startmldonkey.sh

this is necessary so mlnet will take the /Volumes/MAX120/mldonkey/ directory as its root directory.

        19.Jan.2005, 9:42 :: unsorted :: Permalink :: Comments (0)

on my server i like to use some p2p software. my favourite network is the edonkey/emule/overnet network, and since i use mac, i am stuck with the mldonkey client. this works fairly well for me, except that there is a bug which will crash the program (see forum thread on macdk.com), and it becomes unresponsive.
it seams though, that the programm keeps running and downloading even after crashing, but only for a limited time. after this, it will stop downloading.

i did some statistics (see the network activity log entry) to see what median upload and download duration i have:

these stats have been done with mldonkey ver2.5.22 without modifying the file descriptor count. i will redo some statistics with increased file descriptor count.

in the mean time, when testing the not-crashed duration, i add following lines to my crontab:

this causes mlnet to be shut down an restarted 3 times a day automatically. for more information on crontab see this blog entry.

        15.Jan.2005, 13:04 :: unsorted :: Permalink :: Comments (0)

after i did this startup item thing, i soon noticed that it does not work properly:
for some strange reason, the command netstat will be terminated after runnning about 24h.

so, i went back to my good ol' cron jobs and added the netstat command, too:

this means, that any instance of netstat will be killed at the indicated intervall with the command killall, in contrast with kill you don't give a PID as argument, but a name (or also half a name). any process matching this pattern will be killed, so handle this command with care. as an additional security measure you have to have root rights to execute killall.

adding the netstat command to the crontab would have the inconviency that it is not actually startet at system startup, but:

1. i do not reboot my server a lot, actually rather never, so i wouldn't care
   
2. i did not yet remove the startup item. at boot, it is started, and the next midnight it is killed, and the crontab takes over seamlessly

        15.Jan.2005, 12:04 :: mac os x :: Permalink :: Comments (0)

i found a way to start netstat at system startup time. after some google searches to see what is executed at system startup, it found some very useful information on www.kernelthread.com, entitled "Mac OS X System Startup". there i read about /System/Library/StartupItems and /Library/StartupItems, which contains items to be executed at system startup.

after all, now that i have set up everything, i have to say that maybe i would have better done this in /Library, instead of /System/Library.

so here are some excerpts of my terminal to show what i did:

then i modified the file to look as follows:

having this set up analogous to, lets say, /System/Library/StartupItems/CrashReporter/CrashReporter (1), there is another file which seems to be needed:

the presence of this file is necessary, otherwise the startupitem won't be executed, although its contents are secondary.

now the correct properties to make it executable

and leave no traces - the files with the ~ are automatically generated by emacs which creates a backup every time you open a file.

now netstat is launched at startup, you can check it in system.log, look for a message "ConsoleMessage: Starting netstat Network Statistics".
combined with the log file treatment i introduced yesterday, this makes a useful tool for your network statistics.

(1) while you're at it, you may want to disable the annoying crashreporter - just comment out with hash key # the line which says /usr/libexec/crashreporterd and it will not launch at startup any more.

update: see also the information given some days later.

        5.Jan.2005, 16:22 :: mac os x :: Permalink :: Comments (0)

i have been looking for a utility on mac os x which would allow me to track or log my network traffic. after a fruitless search for a program who would do this for me, i got a hint for a command line tool: netstat

after having a look at the man page of netstat i knew that this is the utility i was looking for. now how to organize it so i can record and manage the activities in a useful way?

i need:

• some independency from the terminal, i.e. write logs to a file
  
• netstat should run all the time
  
• logfile should not become too big

solutions i found:
calling netstat with parameter i makes it select the default network interface;
parameter w 60 makes it wait 60 seconds before it prints again.
sudo executes the command as root user.
>> appends the output to netstat.log, where i can access it easily with console.app.
& at the end frees the terminal, so i can use it again for other things.
watch out: if the terminal window is closed, also netstat is gone!

then, to keep this file at a reasonable size, i edited the crontab:
this is the file which keeps information about processes which have to be carried out regularly, so called cron jobs (1). it can only be changed changed with root rights, therefore use sudo.

it contains the following lines now:
to make it easier to navigate in the log file, i print the hour and date every hour.
adding the nice command before the actual command will cause the command to be executed with lowest priority (range is from -20, the highest, to 20, the lowest). for more information see $ man nice.

so this gives a handy solution which doesn't require any human intervention, except at startup i have to run the netstat command manually.

(1) a tutorial for crontab can be found on www.macdevcenter.com, which is actually entitled "learning the mac os x terminal" but in which the functionality of crontab is described pretty well.

update: see also the information given the following day.

        4.Jan.2005, 19:43 :: mac os x :: Permalink :: Comments (0)