Arcanology

For ‘why would you do that !’ kind of things

Webcomics and offline viewing

leave a comment »

A reason to smile during a typical day’s routine is provided when your feed aggregator notifies you of an incoming feed from the webcomicof your choice. My all time favourites include xkcd, PhD comics, The Joy of Tech, UserFriendly, Red Meat and Cyanide and Happiness. Others that make it to the list are Questionable Content, the 5th wave and Dilbert

Now for a late entrant like me into this world, it becomes rather tedious to read the strip online from the beginning. I do not remember where I stopped reading previously and I already have enough bookmarks without needing to add one more. I would like to save them offline so I can read them anytime I want. Then again who wants to navigate every page and save every image.

A simple  customized script in Linux can automate the whole process of downloading . Here I present a Tcl script to download the images from The Joy of Tech

#!/usr/bin/tclsh package require http
proc httpCopyProgress {args} {
    puts -nonewline .
    flush stderr

proc getFile {url i fileName} {
    puts -nonewline “Starting Transfer of file $fileName”
    set out [open $fileName w]
    set token [::http::geturl $url -channel $out -progress httpCopyProgress -blocksize 1024]
    puts “Done”
    puts stderr “”
    close $out
}

# Set start sequence here
set i 1

#Set last required strip in sequence here
set limit 1218

while {$i < $limit} {
    if {$i < 999} {
set num [format "%03d" $i]
    } else {
set num $i
    }
    set url “http://www.geekculture.com/joyoftech/joyimages/$num.gif&#8221;
    getFile $url $num “$num.gif”
    incr i
}

Of course this can also be performed using a shell script / perl / python and the like. The trick in downloading is when you the webcomic issues are not ordered sequentially by their serial number. Rather they url looks something like http://ars.userfriendly.org/cartoons/?id=20090113. Here one solution I can think of is finding the pattern (sequence number / dated)  for the image url and modifying the script to download according to the sequence. If that does not succeed download the entire page, parsing the for the <img> tags for the image url and downloading it. This can also be helpful if you want to add the title of the comic to the filename by searching for the regex in  the downloaded webpage ( I did this for PhD comics). 

If anbody have a better solution I would like to hear of it.

Written by Imad

March 7, 2009 at 7:12 pm

Posted in General, Scripts, Uncategorized

Tagged with , ,

Multiboot OS Installer DVD

leave a comment »

It all started when Windows XP screwed up my partitions. I have three HDDs arranged in this fashion.

IDE1 – Windows (C:) and Other OS’s here

SATA1 – Drives (D:, E:)

SATA2 – Drives (G:, H:)

Normally I hibernate rather than shutdown, and on this particular day I had switched the SATA drives after hibernating. SATA1 became SATA2 and vice-versa.

XP resumed from hibernate and I tried to access the D: drive. Now in an ideal world, I should have seen the contents of G:, but this isn’t one. What I saw instead  were the contents of D: themselves, but trying to access any file resulted in an error. It was the same with every drive. Realizing my snafu, I quickly switched the drives back and rebooted. Trying to access my drives now would result in an error and Windows would ask if I wanted to format the drive. The worst part of the whole thing was drives which could erstwhile be mounted under Linux were no longer mountable. XP had done a great job of corrupting not just the partition table (which can usually be reversed with software) on the drives but also in some way , for lack of a better verb, fubar(ed) it.

Painfully I managed to recover most files using some third party software and decided to reinstall windows since even after formatting the drives, drive G: was still being shown as drive D: (mount points in Windows had been altered), and I wanted to change the drive structure anyway. A single DVD which contained XP with SP3, my flavor of Linux and a few recovery tools seemed to be the perfect way to deal with this situation if  it ever occured in the future. So now I’ll discuss  how I went about doing this stuff.

Much has been written on the method of creating Multiboot DVD on MSFN. But this covers only the creation of Multiboot Windows DVDs. Additionally there is an interesting thread on the MSFN forums which deals with creation of Win+Lin installer/live superdisc. Needless to say, this condensed post deals with my experience of creating the disc by following the steps given in the aforementioned locations. So lets get started:

What you will need ?

i) A CD Bootloader (if I may say so) – I use ISOLINUX (alternatives are CDShell, GRUB4DOS, UBCD, but the menu configuration and other things will be very different. You are on your own with them )

ii) Windows XP setup. I suggest that you slipstream it with the latest service pack. There are lotsa sites out there that describe slipstreaming in detail.

iii) Your choice of Linux distro’s image (can be live/installer). I will be using TinyMe Linux, Slax, Ophcrack, BackTrack Linuxes. Please note that the method will vary depending on the type of distro that you choose. I will discuss the concepts behind it, but you will need to figure out the way to do it for your choice. The thread mentioned above on the MSFN forums contains tips for some popular linux distros

iii) BartPE builder if you plan on adding Windows XP Live to the mix

iv) Some recovery CD like Ultimate Boot CD, which can be of use in diagnostics

v) mkisofs to create the image – Use either the Windows or Linux version depending on your choice of platform. I will be using the command line version

vi) Recommend that you test your DVD image on any Virtual Machine. You can use VMware, Virtual Box, qemu or Microsoft Virtual PC. Virtual Box is my choice of VM. Whatever you choose to use, make sure that you boot the VM from the multiboot iso image that you will be creating.

vii) A hex editor

viii) Patience, persistence and lots of it. Things have a tendency to take time, behave unexpectedly or simply fail and make you miserable. It took me three weeks and at least 3-4 hours per day to finally customize it the way I wanted.

ix) Brains – This is a guide for people who love to experiment. I may be wrong or you may be doing things wrong or things just go haywire. Either way you will have to figure it out and move ahead.

CreateDisc()

1) Create a work area

Create a folder for your work area. Let’s name this DVDROOT

2) Windows XP Setup Layout

You need to do this from within XP.

Very Important ! First create a copy of boot.ini. This file is in your root directory and is usually hidden and a system file. I take it you know how to view such files

Create a directory DVDROOT/Setup/XP. Copy the contents of the Windows XP CD here. This contains the directories {cmpnents, DOCS, I386, SUPPORT, VALUEADD } and a few other files

Now we need to obtain the XP setup boot files. To do this type the following in the startmenu run box:

<path to DVDROOT>/DVDROOT/Setup/XP/I386/winnt32.exe /noreboot

Now we need to obtain the XP setup boot files. To do this type the following in the startmenu run box:

Create a directory DVDROOT/Setup/XP. Copy the contents of the Windows XP CD here. This contains the directories {cmpnents, DOCS, I386, SUPPORT, VALUEADD } and a few other files

Very Important – First create a copy of boot.ini. This file is in your root directory and is usually hidden and a system file. I take it you know how to view such files

You need to do this from within XP.

This will bring up XP setup.

  • Chose Fresh installation, and proceed
  • Do not choose to download updated setup
  • Click on Advanced options and make sure that “copy all installation files from CD” is checked or grayed out

Now setup will create two hidden folders in the root of your main drive (usually C:).

Delete the folder named $WIN_NT$.~LS.

Rename the folder named $WIN_NT$.~BT to say PRO1 (or any four lettered word) and move this folder to DVDROOT.

In DVDROOT/PRO1, delete BOOTSECT.DAT, migrate.inf, winnt.sif. (you can modify winnt.sif to customize and automate your setup. Check out the web for instructions on doing so. You can skip deleting this file if you choose to do so)

Important ! – Running XP Setup modifies your boot.ini. So replace the modified boot.ini with the copy you had made earlier. If you haven’t done so, check the web on how to create one.

In your hex editor, open up DVDROOT/PRO1/setupldr.bin. Replace all occurences of i386 with PRO1. There should be 4 occurences. Save your changes.

Open DVDROOT/PRO1/txtsetup.sif in any text editor. Change SetupSourcePath to the directory we copied XP CD to. In our case

SetupSourcePath = “\SETUP\XP\”

Create the boot sector – Download the file PRO1.zip. You can also use a program like ISOBuster or UltraISO to extract the bootsector from the bootable XP cd. Extract the file PRO1.dat edit in the hex editor. Replace the occurence of I386 with PRO1. Save your changes and rename the file PRO1.bin

Copy (don’t move) the following files:

DVDROOT/Setup/XP/readme.html

DVDROOT/Setup/XP/WIN51* , where * is a wildcard

to the DVDROOT/ folder

Note: There are plenty of files on the Windows CD which are not necessary for normal install. You can safely remove them, freeing up additional space on the disc. Check out the web to see which files/folders in the CD can be safely removed.

2a) Windows XP Modding

It might be worthwhile to customize the setup of Windows XP to suit your own needs. Modding the setup allows you to :

i) Add your drivers to Windows XP during setup itself

ii) Add certain applications for autoinstall with XP

iii) Add the latest windows update hotfixes to the setup allowing a patched system soon after install.

iv) Automate the serial key, region, password and networking settings so as to avoid typing anything during setup

v) Customize the look and feel of your OS

This is too diverse a topic to be dealt here. Check out the unattended xp install guide at msfn to do the same. This in my opinion is the best guide out there to do such things.

Personally in my DVD, I have WinXP original and WinXP modded both. If you need to add the modded system, follow pretty much the same instructions as given for Windows. You will just need to create a separate folder to add the setup files (say PRO2) and copy the modded CD contents to a folder like DVDROOT/Setup/XPMOD/ and modify DVDROOT/PRO2/txtsetup.sif to change the necessary paths as mentioned earlier.

3) Add Windows XP Live

This allows you to run Windows XP from the CD itself without installing it much in the same way as lots of Linux distros out there do. Check out BartPE builder to generate your own live XP cd. Here we are concerned with the aftermath.

Tip: Using BartPE builder you can also add your own applications to Windows XP live. A carefully chosen selection of applications might enable you to build a recovery system with your own set of tools!

Once the live CD is created, copy its contents to the DVDROOT/ folder. Rename I386 to XPPE. Now open XPPE/setupldr.bin with a hex editor and replace all I386 with XPPE (4 occurrences), as you did earlier and save the file. Now open XPPE/txtsetup.sif and change the entry for SetupSourcePath. Now,

SetupSourcePath = “\”.

Now copy the file ERD1.zip. Extract the file ERD1.dat and open it in a hex editor. Replace I386 with XPPE and save. Rename ERD1.dat to XPPE.bin

At this point Your DVDROOT tree should look something like this

Directory Structure

Directory Structure

4) Adding Linux

This part can be a bit tricky for those of you not familiar with Linux/shell scripting and certain commands in some cases. You might also need to get a few things done from within Linux. This is also the stage offering a lot of scope for experimentation.

The general idea is this. To boot linux from a CD, a tiny linux kernel and an initial ramdisk are used by ISOLINUX. Once the kernel is loaded, certain there is an init script which will later uncompress/mount rest of the system on the fly. So all that needs to be done in most cases is to pass appropriate paths to the kernel in ISOLINUX configuration file. In cases where you might need to alter the directory structure to fit it into the disc, you will need to extract the boot scripts from the ramdisk image, modify them and recreate the ramdisk image.

I will discuss adding the Slax distro to the DVD. Adding other Linuxes may be alike or involve some hardwork. The first thing you do is examine the isolinux.cfg file in the bootable CD of your distro. It is usually in  a directory named isolinux, and can also have other filenames such as menu.cfg. Only way to know is to open the files in some ASCII editor and see for yourself. In the isolinux.cfg file, you can know which files comprise the kernel and the ramdisk. Coming back to Slax, copy the folder slax present in the Slax live CD to DVDROOT/ . Create a folder isolinux under DVDROOT. Now from the boot folder in the Slax live CD, copy vmlinuz and initrd.gz to isolinux. You can rename them to something like vmslax and initslax.gz respectively so as to differentiate them if adding other Linux distros later.

For adding something like Fedora Live, refer to the thread on msfn mentioned earlier. The procedure is more complicated and involves editing the startup script to reflect your changes.

5) The boot menu cometh

This is the stage for the artistically twisted to exemplify their flamboyance. Let me retract a tad on that, after all this is not a Photoshop tutorial. But this is where you add graphics and colours and menus. I will not cover all those in detail here. Suffice it to say, I will just help get started on writing a basic menu. Play around with it if you want to show-off some swell menus.

Go to the directory we created – DVDROOT/isolinux. Here create an ASCII file. Use an editor which supports UNIX format  (new lines begin on <LF>, on Windows it is <CR><LF>), or I cannot guarantee the results. On Windows, Notepad++ supports this feature; on Linux, just start typing on anything of your choice (Mac fanbois shouldn’t even be looking at this tutorial. They all seem to be on 7th heaven of Desktop Operating Systems. Lets not disillusion them or they all start flaming).

My condensed menu looks something like this

DEFAULT vesamenu.c32
TIMEOUT 50

MENU HIDDEN
MENU AUTOBOOT Booting from hardrive in # seconds. Press any key for options …
MENU BACKGROUND splash.png
MENU TITLE Choose your Operating System

LABEL local
MENU LABEL ^1. Boot from Hard Disk
LOCALBOOT 0x80
MENU default

LABEL winxp
MENU LABEL ^2. Windows XP Professional
KERNEL /PRO1.BIN
LABEL winxpmod
MENU LABEL ^3. Windows XP Professional (Modded)
KERNEL /PRO2.BIN
LABEL winxpe
MENU LABEL ^4. Windows XP Professional Live
KERNEL /XPPE.BIN

MENU BEGIN ^5. Slax
LABEL Parent
MENU LABEL ^0. Back to Main Menu
MENU EXIT

LABEL Slax1
MENU LABEL ^1. Slax
KERNEL vmslax
APPEND initrd=initslax.gz ramdisk_size=6666 root=/dev/ram0 rw autoexec=xconf;telinit~4 changes=/slax/
MENU END

You can use MENU BEGIN and MENU END to create submenus. A caret (^) symbol before a character in a label acts as a keyboard shortcut. Here we can see that for Slax linux, we essentially pass the same arguments to the kernel as in the Slax live CD. Depending on the directories under which that you have organized any other Linux, you might need to create an entry in the menu and pass the appropriate arguments to the kernel using APPEND keyword.

Download syslinux package, and go through the help files to understand how you can add more frills to the menu system. From syslinux or from the isolinux directory of any other live CD, copy boot.cat, isolinux.bin, memdisk, vesamenu.c32 into DVDROOT/isolinux/ directory. You can also copy a bmp/png 640×480 image and add it here to act as the background to your menu using SPLASH.

The splash menu I added

DVD Boot Splash Screen

DVD Boot Splash Screen

6) Create a disc image

mkisofs -o <output iso filename> -V “<volume name>” -J –N  -D -joliet-long -no-emul-boot -boot-info-table -boot-load-size 4 -b isolinux/isolinux.bin -c isolinux\boot.cat <path to DVDROOT>

I used the above command on Windows (yes, it is isolinux/isolinux.bin) . On Linux you can use the appropriate format for paths.

Very Important ! Standard DVD size is around 4.4 gig. But make sure that the contents of your DVD are under 4 Gig otherwise Windows XP setup will just croak and you will be none the wiser as to what happened. I found this out after a lot of trial and error

7) Test it on a Virtual Machine

Try out the DVD image on a virtual machine by booting the machine using the image as the first boot drive. Most VMs should allow this. As part of testing, I suggest you completely install Windows XP and boot Windows in the VM.

8) Burn DVD burn!

Credits

This is just a condensed version of the material I found on the web, especially on flyakite.msfn and the forums. Credits go to the original authors and users on the forums whose suggestions enabled me create this.

Written by Imad

February 3, 2009 at 4:07 pm

Hello wordpress!

This is  a blog where I plan discuss things esoteric (the url is indicative I spz). This shall vary between diverse topics intended for the geeks and übergeeks with whom we share this planet.

Posts will be tech related,  rants of mine, a bizzare idea swimming in my grey cells,  a project which I carried out or whatever  I feel will justify the few kilobytes it will occupy on the server disk.

Look forward to updates at haphazard intervals, hackneyed phrases and the like

Written by Imad

December 17, 2008 at 6:55 am

Posted in Uncategorized

Follow

Get every new post delivered to your Inbox.