Wednesday, April 06, 2005
I know it's been a while, my apologies, nothing but a very geek/tech related post today. This one's on a happy note. I solved a little mystery that was really getting under my skin. It wasn't even important but it really opened my eyes about a little quirk with Unix file systems (and filesystems in general).
I've never really made much distinction of the notion of how files often take up more space on disk than their actual filesize. Well, they do, mostly because files must be in units of blocks (or clusters, same thing, different term for different filesystems). Often filesizes don't fall on even block bounderies so there's a tiny chuck of wasted space on the end of a file which represents the unused portion of a block (this is known as "internal fragmentation"). My ext3 partition use a 4k block size, so for every file, there's the potential to wast 4K minus 1B worth of space.
I've know about this for some time now. What I didn't know, is that this isn't the only space that's "wasted" (wasted is a harse term, "necessarily used for addressing" would be more appropriate) on Unix filesystems.
I found this out by taking the block count given to me by the stat command, and multiplying by 512 (stat always reports blocks in units of 512). I figured this was how one calculated actual filesize on disk. The filesizes were different alright. For very small sizes, the difference in size was always smaller than my block size (4k). However, larger files had a much, much larger difference. It wasn't phenominal (1.3 megs difference for a 1.4 gig file) but it was perplexing, because I didn't have an explanation for it. Why wasn't the filesize difference fitting within a block?
The answer took me a while to find out. I hit the chat rooms on freenode like a banshee but no one had an answer. I researched and researched on the web on ext2/3 and Unix filesystems. I finally found out why. It has to do with the way inodes address data blocks. Basically, the first 12 blocks are addressed directly. This makes addressing and reading the firts 12 blocks (always the most important in most files) of a file very fast. However, the pointer after that isn't a direct address, it's a pointer to another block that can address (1024 * block size) more data. This is known as the single indirect block. After that you have one more pointer. This one points to a pointer that points to several blocks that point to data, it's known as the double indirect block. This basically lets you address around 4 gigs or so worth of data. Finally if you need more than that, there's the third indirect block (you get the idea for this one) that'll let you go up to 2TB worth of data (the filesize limit). So basically, at 13 blocks, an extra block is used to store an additional 1024 * (block size) data. After 1037 blocks, 2 additional blocks are used to address data, and another block is used for every 1024 blocks additional data stored after that, until 1024 * 1024 blocks have been stored, then the number goes up by 2 instead of one, then 1 for every 1024 blocks, on and on until you start using that third indirect block, at which point I believe it'll go up by 3, etc. Basically, the filesystem grabs additional blocks for addressing as needed, as the file grows. This makes it so inodes only have to be a block in size instead of large enough to have data that addresses every block in a file (would waste lots more space let me tell ya). These address blocks are reported by stat as being used by the file.
In conclusion, with a large file you have a tendancy to use (very, very, roughly anyway) an additional 0.1% of the file's size in address blocks. FAT16/32 and NTFS, etc. have similar ways of using additional disk space for addressing, however when mounting them under Linux, stat won't report these used blocks. Kind of makes it look like fat16/32 and NTFS are doing a better job preventing wasted space ;-) (I know, I know, the space isn't "wasted" it's used necessarily for addressing, but it looks wasted to the layman). Not true, but it sure looked that way to me for a while! I'm glad I finally managed to figure this problem out for myself.
And if any of this is wrong, please someone email me and straighten me out. I'd really love to understand this correctly.
Friday, March 18, 2005
'nuff said for today :-). Well, that and Maggie and I are seeing Jay Leno tonight at Cache Creek.
Thursday, March 10, 2005
Again, over an hour for one, friggin' command.
I'm just tired of it. Again, I'm blogging this so I never forget it. In fact, I'm gonna fill the beggining of this post with keywords so that Google maybe pays attention to it and helps someone find the exact information I needed. Linux, Fedora Core, LVM, Logical Volume Manager, Rescue, Linux Rescue, Mount, Mounting LVM. There.
First, a little background for the readers.
Today, I installed Fedora Core 3 on one of my work machines. The install went smoothly, I decided to use the logical volume manager to setup the partitions since there were two 120 gig drives in this particular machine. I made a /boot outside the LVM, then a /, swap, and /home within one Volume Group, spread across the two hard drives.
The machine booted, I then installed Nvidia's nforce on-board nic drivers [Edit: I found out later that apparently Fedora Core 3 actually comes with these, very nice] (binary only, but hey, they work well) and all was good, I then went to perform an up2date in KDE. I discovered to my horror, I'd been way too conservative with the root partition and there wasn't enough space to install all the updates.
The next hour and a half or so demonstrated my severe lack of experience with LVM and Linux file systems.
Attempt 1: Boot with Knoppix and see if QTparted will do the trick.
Result: Utter failure and discovering, to my horror, that Knoppix has zero support for LVM. Note to self: lookup remastering my own Knoppix CD with LVM2 and device mapper support.
Attempt 2: Hit the Chat rooms
Result: A little bit of knowlege is dangerous. My first question in #fedora on freenode.org was met with "Read the LVM Howto". So I skimmed it. I asked more questions. That's when someone mentioned the command lvresize. Hmm, okay, I think, I do a "man lvresize" and start reading away. After about 30secs-1min I figure, Okay, if I'm reading this right, and if what the guys in the chat room are telling me is right (I can resize volumes while booted), I should just be able to do the following commands:
lvresize --size -4096M /dev/VolGroup00/home
lvresize --size +4096M /dev/VolGroup00/root
Which I did, and both seemed to work flawlessly. I then triumphantly said how much I love LVM in the #fedora chat room, and told them what I did.
To which effect, one of the smarter folks in the room put me in my place by messaging me and saying "Did you shrink your filesystem first?"
To which I stupidly and truimphantly said, "No, how do you do that?"
To which he classically shot back, "Consider /home fried".
And fried it was upon a reboot and an attempted forced fsck.
Attempt 3: Rebuild /home, grow / to fill the increased logical volume size.
Result: Googling for hours to find out how to work lvm. I was eventually successful. /home was an easy fix; thankfully, the system had just been installed, so I had absolutely nothing of value on /home. I just ran a mkfs -t ext3 /dev/VolGroup00/home, then mkdir /home/sam, cp -a -r /etc/skel/* /home/sam, chown -R sam.sam /home/sam. It was a good thing I hadn't destroyed /! However, growing the / partition to take advantage of it's new space proved to be a painful chore. While booting into the rescue portion of the Fedora install, it's wizard offered to mount root for me. That was great, except that I couldn't resize / while it was mounted and because of unknown file activity, the rescue CD wouldn't let me dismount it. So I booted up again this time skipping the root mount portion. Now I was in unfamiliar territory, with absolutely no knowledge of how to mount LVM partitions. After much googling, the following did the trick.
(from within the LVM prompt)
> vgchange -ay VolGroup00
Then I was able to finally access /dev/VolGroup00/root without it being mounted. After a e2fsck -f /dev/VolGroup00/root, I was able to peform a resize2fs -p /dev/VolGroup00/root 7G successfully.
I was absolutely exhausted and frustrated by the whole experience. It is often that people within Linux chat rooms with tell you to RTFM. I really don't mind this response much, but seriously, sometimes I feel that I can't possibly be the only poor sap to have run into this kind of stumbling block, and that the HOWTO's would be so much more useful if they were written in a more "cookbook" style (a la O'Reilly) with gads of examples. As it is, they seem to prattle on, and one has to cut through sections with a machette before one finds what they're looking for.
</end rant> (again)
Monday, March 07, 2005
Why God?? Oh Why Does Backspace or Delete Sometimes Not Work On Terminals?
I know my last entry said that was it for today. I lied.
After having worked with different Unix flavors for over 6 years I'd like to think that that I'd have this problem down pat. Yes, I know why it exists. There's a thing called ASCII. There's lots of redundant codes that do the same thing (BS vs DEL, CR vs LF) and at one time different computer vendors picked one instead of the other. Chaos has ensued ever since. But sometimes, just sometimes I think it's a bad fucking excuse and the whole damn thing should have been solved by now by always making BS and DEL do the exact same thing in every single fucking program!!
I lost an hour's worth of work trying to solve this problem (again) tonight with GNU Screen. I love GNU Screen, but it's fallen a notch in my eyes tonight. Some programmer somewhere is going to say it was my fault. And I say to that very programmer, for my own carthartic benefit, "FUCK YOU ASSHOLE!" There, I feel better all ready.
I've come to solve the backspace issue by making sure my terminal erase is always "^?" (That's ASCII character DEL in Keyboard Access syntax, character 127 in decimal, 177 in octal and 7F in hexadecimal. It's scary to think I've dealt with this issue so much that I can recite that without a second thought). I use DEL because Emacs is my editor (another GNU program, one would think that would play in my favor) and that program has backspace issues which took me a while to solve. Used to be GNU Emacs ignored stty's erase setting, so one had to hack emacs to behave based on how your terminal emulation was layered. Finally the emacs folks fixed things so that it paid attention, but it was too late, I'd already gotten used to using terminal emulation software that would let me program it so it sent "^?" for backspace. It's better this way anyway, if "^H" is bound to erase in emacs, one can't use the online help system through C-h, and I've quite gotten used to that.
Enter gnu screen.
I've used screen on and off. I've experienced BS issues with it, to the effect it often seems to be programmed to want "^H" to be erase. I've been lazy about getting around it in the past. Not tonight. I was hell bent on solving the problem.
One hour later, I exasperatedly found this remedy. I put the following in my .screenrc:
bindkey -d -k kD stuff "\177"
A whole goddamn hour for the above line. I hit a lot of fucking web sites for that shit.
Aparently screen completely ignores your terminal erase setting (set via stty) and picks its own based on your TERM environment variable. Aparently the TERM variable I was using for Mac OS X Terminal.app (xterm-color) doesn't use ^? it uses ^h for erase. And screen wasn't gonna have it any other way. Unless I changed my TERM variable. However, I specifically use xterm-color, because I find ANSI colors frequently do not work in Terminal.app when I'm not in screen unless my TERM variable is set to this.
Would it have killed the screen developers to make screen pay attention to the terminal erase setting? Would it have hurt them at all? I would have called it behaving as advertised and it would have saved me loads of frustration. Do what I want dammit! Do what I want! Argh argh argh argh!
[Edit: Guess what, after all this fiasco, I ran into another backspace problem again with mc within screen. Had to fix that one too. I'm so happy I could cry, I feel the tears coming now...]
NewOrder - computer security and networking portal
Just fucking hilarious.
And that's it for today ;-).
Wednesday, February 09, 2005
So I Met Maggie's Parents Last Sunday...
...and believe it or not, everything went very smoothly, a little akward, but smoothly. It was certainly no Meet the Parents. Time will tell what they think of me, I hope I made a good first impression. Maggie seems to think I did all right. I probably should have taken some pictures, but we all know how I am about that.
Friday, January 21, 2005
Nintendo's Censorship History
You never know what you'll run across while surfing Wikipedia. I was browsing the Nintendo page discussion node and came across this site: http://www.filibustercartoons.com/Nintendo.php. It documents a formal policy Nintendo had on censoring content in its games, before the ESRB rating system was devised. Kind of an interesting read for the video game folks.
Sunday, January 16, 2005
Maggie and I Hit Lakeport
On a whim, Maggie and I decided to finally do that Lakeport trip I'd been promising her. We had a great, if short, time, not a lot's changed in the old place (they got a blockbuster video! Who'da thunk!) still it was nice, I showed Maggie around downtown, the Museum, the Book Stop, TNT's on the Lake, even my old place (just the outside) and my schools. Ah memories.
Incidentally, I guess Jay Leno's gonna show up at Cache Creek sometime in March. Man that place has gotten huge.
I'll try and have some photos posted up one of these days. We all know how I am about that however ;-).