Posts
I'd have said "not at all" but maybe I'm wrong...it always seemed like *I* was the one with difficulty...does it take an Advanced Degree (tm) to figure this out?
[sorry about the links having been wrong, I didn't realize Blogger was prepending the http part]
British glamour model Lucy Harrold apparently has trouble. Enough so that despite being on a UK "dating show" called "Take Me Out", and having adequate finances anyway, she created her own singles website to help solve this.
Seriously?
OK, I don't actually like the shape of her face, but certainly from the neck down is more than ok (although her tits are fake, eyebrows too)...I wouldn't have been interested...and of course she wouldn't be interested in me. But if she'd asked me (ok, the odds there are actually negative), I'd have sure had ONE date with her.
Is it that difficult for them?
So what is she looking for?
what are ANY of them looking for?
Brad Pitt, apparently. or Ben Affleck, it seems.
Excitement.
I did another blog on this a ways back...
Apparently Jennifer Lawrence has trouble finding a date, too.
Seriously? OK, she's ridiculous famous, very attractive without it being ridiculous. She has a lot of money, fancy house...but no dates? Granted, under those circumstances you can't just casually meet people who aren't either sycophants or wanting something themselves, but it shouldn't be impossible. Of course who you DO meet is going to be other people who are in the same line of work and are likely to be more than a bit self-centered. So we're back to the question of "what does she want?"
Which was Freud's old question, too.
---
OK, anecdotes are not data, but we all have some anecdotes. Here's one:
http://www.huffingtonpost.com/xojane-/online-dating_b_5909274.html
She leads with the photo. I'm sure her friends say she's marvelous...but she leads with the photo, and you can tell what's going on. She's overweight. Precisely the thing your friends won't tell you. Of course she kinda knows, you really do always know when you're overweight, altho maybe you can't admit it to yourself. I know; I'm overweight right now, although down 10 from a year ago (I have a dog now; a dog for whom a "short walk" is minimum half a mile, 4X/day). "many ramen noodle dinners" she says, but obviously that hasn't helped, she's been putting too much bacon in them. "What was turning them away? Was it my looks -- which was based on the best photos of me?" No, hon, it's the weight. And if it was those same photos, eew; I'd include a pic of me with my sailboat (Hobie 16) years
ago (and mention that it was an old pic), at least that's something a bit more active.
Well. She has gotten to experience what most of us guys go through over and over: "the rejection thing". She certainly seems surprised. "after two months I had been rejected by countless numbers of men" -- aaawwww. Two whole months. I stopped after 10 years.
OK, so we're shallow. Sorry. Lose some weight. Several comments say this, followed by "her weight is fine" except that it's not. I guarantee that a 25-lb weight-loss would turn things around pretty quick for her. OK, maybe 40. (also: go read this,
and look at the pics--WAY better than the two she showed for her
article/"profiles"; which had that "desperate" sound to me, altho apparently she
has clinical depression--that guarantees trouble here)
And as some of the comments suggest, she needs a better self-marketing pitch. It's not like going to Comicon, to a presentation on your fave comic, and know that everyone in the room has a common interest, so you can pick a guy whose looks you like, know he's nerdy and shy and talk...
another great comment: "not all girls want a ben affleck type."
Of course they do. What would they do if Ben Affleck showed up at their door and said "come have dinner with me" ? They sure wouldn't slam it in his face...it'd be more like "Give me 5 mins I need to change clothes first...how fancy?"
Actually, she needs a different approach. My best experience at her age was with a church singles group a cousin suggested I attend. That group had ~200 people attending, all roughly 22-32, and as I recall there was some couple there saying they were engaged at least once a month, maybe even once a week.
---
I'm sure you've heard the words about them having a hard time, because the easy way tended to attract the "wrong kind of guys". I argue that that really means they aren't attacking the problem pro-actively.
Anecdote: One of my best friends met his wife at a bar. Yes, really. 1989. Been married since '90. That *can* be done, but you shouldn't expect it.
Being a nerd doesn't preclude success here, but it does argue for an alternate approach. You need to get involved in a group (or more than one) that has MOTOS and a specific focus you are interested in, and then actively participate.
The better looking you are the less difficult this is overall, so you have to be really honest with yourself about this, and deal with the fact that there's a correlation.
for comparison: I'm about 15-20 pounds overweight. I'm tall, but no more than average looks. I'm "difficult", so few-to-none people like me. I'm now approaching 60, and retirement (see blogs on that topic). None of that is fatal, but it's not helpful. I forget who said this in a movie/tv-show, it was a early-teen male: "my superpower is being invisible to women" -- applies to me -- they can only see me for collision-avoidance purposes--and that's always been the case.
A few more little anecdotes (which still aren't data):
Guy I knew at college ("K"). Civil Engineering. "K" was overweight when I knew him (280, I asked once). That basically killed his having a date, despite his being one of the most likable people I've ever known. I think he had one date the entire 4 years, and that was because I suggested her to him. Something like 12 years out of college he meets a woman at the golf-pro shop where he plays golf. She does too. He's still the same shape, but with a lot less hair. She's about the same shape, too. They were married ~15 years; he died young, from cancer; one child.
Woman I knew very slightly at work 30 years ago: I only had a conversation with her a couple of times (let's say her name was "G"). Not attractive. A bit overweight. Pleasant enough, intelligent enough, knew how to dress...but the looks were a severe handicap. No dates. Eventually she gave up; I recall a little bit of conversation with another woman ("M", foreign born, distinctly more attractive, I sat next to her for 6 months) at work about it, because "M" told me that "G" had said she was done trying/hoping. (I hope "G" did ok with her life; "M" got married in 1989--almost coulda been to me.)
Woman I've known a while: bought herself a Jaguar convertible car in 2010. All of a sudden she was getting new attention from random men, because of the car.
Woman at work two years ago: stunningly attractive. Attractive enough to have to fight guys off with a club just to get to her car. I only talked to her 2-3 times ever; I know nothing of her social life, but she could have a dinner date with a new guy every single day for years if she wished.
Man. It's a wonder most of us ever get together.
Thursday, October 09, 2014
Friday, August 29, 2014
Retirement
Late in 2013 my job situation was getting kinda weird. There was contracting funny-biz at work, on-again, off-again, on-again, off-again...yeesh.
By TG it looked ok again, contract re-awarded...but my tasking wasn't in it, apparently, and that started to go weird in Jan, on/off/on/off--bam! Not going to have a job on Feb 1. What to do?
And then for the first time ever, "Retirement" was now on the list of options. Not *quite* ready to do that (well, I AM, but it ain't quite time, need to wait about 3 years).
Changed employers very abruptly, stayed in exactly the same job slot, that's good for about 2 years it looks like, so that's now the retirement planning horizon. There are a couple of circumstances whereby it might happen sooner, but I think they're unlikely.
Personally, I can hardly wait...man I'm tired of the rat race around here. Job is more interesting than it was six months ago, but I am getting hammered by this ridiculous database we are using (I've found two seg-fault-fatal crashes in in the last two months, and the indexing isn't anything like SotA speed [a few weeks later: now we have damaged data because deletes are problematic]), and that is getting tiresome (although my next task there will be to try to do a complete replacement for the indexes, Berkeley DB Java is ~4X faster, and I think that will do the job; the C version is probably twice that speed, but I don't know about a Java interface, and I dislike horsing around with JNI).
Got plenty to do when I do retire--enough to keep me busy until I die. Rereading my books is a minimum 10-year project. Working the model railroad runs forever. Computer games never end, programming never ends. Multitude of projects on The Ranch to do, several of which are engineering experiments.
That's all probably good for 30 years, at which point I'm probably getting a bit feeble, and probably ought to just go ahead and die anyway.
It can't start too soon!
By TG it looked ok again, contract re-awarded...but my tasking wasn't in it, apparently, and that started to go weird in Jan, on/off/on/off--bam! Not going to have a job on Feb 1. What to do?
And then for the first time ever, "Retirement" was now on the list of options. Not *quite* ready to do that (well, I AM, but it ain't quite time, need to wait about 3 years).
Changed employers very abruptly, stayed in exactly the same job slot, that's good for about 2 years it looks like, so that's now the retirement planning horizon. There are a couple of circumstances whereby it might happen sooner, but I think they're unlikely.
Personally, I can hardly wait...man I'm tired of the rat race around here. Job is more interesting than it was six months ago, but I am getting hammered by this ridiculous database we are using (I've found two seg-fault-fatal crashes in in the last two months, and the indexing isn't anything like SotA speed [a few weeks later: now we have damaged data because deletes are problematic]), and that is getting tiresome (although my next task there will be to try to do a complete replacement for the indexes, Berkeley DB Java is ~4X faster, and I think that will do the job; the C version is probably twice that speed, but I don't know about a Java interface, and I dislike horsing around with JNI).
Got plenty to do when I do retire--enough to keep me busy until I die. Rereading my books is a minimum 10-year project. Working the model railroad runs forever. Computer games never end, programming never ends. Multitude of projects on The Ranch to do, several of which are engineering experiments.
That's all probably good for 30 years, at which point I'm probably getting a bit feeble, and probably ought to just go ahead and die anyway.
It can't start too soon!
Sunday, August 24, 2014
One last time playing Dungeon Siege
One of my favoritest games. I've been through it end to end probably six times, as all four specialties. Figured I'd do one last time, try something I've wondered about since the beginning.
At about the final merchant, there's a spell called "Bomb". You have to have Combat Level 60 to use it, and You can't reach level 60 until about that point, or maybe later (ok, that makes sense), but that's really only if you have one character do ALL the work, using ONLY Combat magic.
Otherwise, you pretty much can't reach level 60 at ANYTHING.
"Bomb" is "Explosive Powder" on steroids, about 3-4X the power. I'm very fond of Explosive Powder, but since it maxes out at 70 pts, it's mostly only useful for getting the attention of an opponent from a distance to separate it from a crowd, or do some special sniping. You'd want Bomb to do the same (which it appears to), but available sooner, like maybe level 45 or so--60 is close to useless, because it's almost the end of the game before you can use it.
Mostly what you want to do is have really good summons to help you. That's the most effective way to go, because their health is be 2-3X you team.
If you do melee players, you most want to have freeze weapons. That, combined with summons is great--your team is maybe 16 players, half have freeze weapons, which means opponents basically can't hit back while you just pound them.
Also interesting: why I didn't notice this before I do not know: when you have to fight Gresh, there are those bunch of stone columns that appear out of the ground. I thought you were locked into that little space with him, but apparently that's not true, you can run back between them where you came from, or off to the right where there are lots more opponents (not a good idea). First time through I could not complete this, had to run the cheat code for invincibility. But when you can retreat safely, it become reasonably easy...I had no idea, I thought it was a locked arena.
At about the final merchant, there's a spell called "Bomb". You have to have Combat Level 60 to use it, and You can't reach level 60 until about that point, or maybe later (ok, that makes sense), but that's really only if you have one character do ALL the work, using ONLY Combat magic.
Otherwise, you pretty much can't reach level 60 at ANYTHING.
"Bomb" is "Explosive Powder" on steroids, about 3-4X the power. I'm very fond of Explosive Powder, but since it maxes out at 70 pts, it's mostly only useful for getting the attention of an opponent from a distance to separate it from a crowd, or do some special sniping. You'd want Bomb to do the same (which it appears to), but available sooner, like maybe level 45 or so--60 is close to useless, because it's almost the end of the game before you can use it.
Mostly what you want to do is have really good summons to help you. That's the most effective way to go, because their health is be 2-3X you team.
If you do melee players, you most want to have freeze weapons. That, combined with summons is great--your team is maybe 16 players, half have freeze weapons, which means opponents basically can't hit back while you just pound them.
Also interesting: why I didn't notice this before I do not know: when you have to fight Gresh, there are those bunch of stone columns that appear out of the ground. I thought you were locked into that little space with him, but apparently that's not true, you can run back between them where you came from, or off to the right where there are lots more opponents (not a good idea). First time through I could not complete this, had to run the cheat code for invincibility. But when you can retreat safely, it become reasonably easy...I had no idea, I thought it was a locked arena.
Thursday, March 20, 2014
Local weather. I hate snow.
When I was a kid I lived in cold places several times...one of them was Colorado. Right after that I lived in Hawai'i.
I like Hawai'i. I like that kind of weather.
Now I live in Virginia. Mostly this is ok.
So this year we got four feet of snow in my area.
FOUR FEET OF SNOW.
That is why I went to Texas out of college, as opposed to Syracuse.
I hate snow.
I like Hawai'i. I like that kind of weather.
Now I live in Virginia. Mostly this is ok.
So this year we got four feet of snow in my area.
FOUR FEET OF SNOW.
That is why I went to Texas out of college, as opposed to Syracuse.
I hate snow.
Wednesday, March 19, 2014
Divinity 2 game
I replayed most of Divine Divinity first, but gave up when I kinda got lost about what things needed doing next...
Well, D2 has somewhat the same issue...game is visually good, but it's got a large-ish flaw about managing your quests and the locations you have to go to to complete them. I repeatedly had to refer to online sources as to where things were. I liked the "flying fortresses" best of all, they reminded me of Oblivion gates.
Too often the combat devolves into a click-fest, however, and that is not something my hands are capable of on a large basis.
Getting to be the dragon was interesting, esp once I realized I had power-ups I could take, and that "fireball" was actually a homing missile.
I haven't finished, and probably won't. It's clear my attack finger is not up to the next battle I have to fight...
ash, well, I have plenty to do with DLC in Skyrim, and none of that is a major click-fest.
Well, D2 has somewhat the same issue...game is visually good, but it's got a large-ish flaw about managing your quests and the locations you have to go to to complete them. I repeatedly had to refer to online sources as to where things were. I liked the "flying fortresses" best of all, they reminded me of Oblivion gates.
Too often the combat devolves into a click-fest, however, and that is not something my hands are capable of on a large basis.
Getting to be the dragon was interesting, esp once I realized I had power-ups I could take, and that "fireball" was actually a homing missile.
I haven't finished, and probably won't. It's clear my attack finger is not up to the next battle I have to fight...
ash, well, I have plenty to do with DLC in Skyrim, and none of that is a major click-fest.
Wednesday, January 15, 2014
Dec/Jan gametime
A while ago, GOG (www.gog.com) offered Sid Meier's Alpha Centauri as a download. This was a favorite a while ago, so for $5 (or was it less?) that was an easy decision.
I'd forgotten how challenging SMAC could be...intellectually the hardest game I've played, because of the sheer range of variability it can have. It's a mostly 2D game.
There's a randomly generated world map; there's a pre-created standard one (actually, probably several). Maps can have several sizes, S/M/L/XL. They can have a variable amount of ocean surface. Land surface is random. There are seven skill/difficulty levels. There's an R&D tech tree, advances are mostly random. Your starting location on the map is random. Ground resources are random. Mobile units that you can create have near-infinite variability of designs that you choose if you wish.
There are seven playable factions; you play one of them, and they're fairly different, different strengths and weaknesses.
If you go with a large map and lots of ocean, the factions are (probably) on islands separated by a fair amount of water. This allows unfettered development with little or no conflict. Then success is based on how fast you expand how far.
Wow. Not easy at all. I had to relearn all the detail. I've only ever played as two factions, green (default) and white. And I've only played two skill levels (default and +1).
I need to try some of the other factions and harder skill levels, now that I've been through it again.
-----
A month ago I received a URL for something on Youtube...it was the opening sequence from Skyrim. You know the one, you're about to get your head chopped off, when all of a sudden, THE DRAGON appears overhead, and you escape in the confusion.
That's the normal situation.
This link was to a machinima of how someone made a game mod to change it. This designer happens to like trains, steam engines in particular, so he's made several mods involving them. There's a spell that rains trains from the sky--do that in town and pretty soon you're Public Enemy #1.
But this startup movie. He's replaced the dragons throughout the entire game with trains. So when you should be hearing the dragon roar, you actually get a train whistle.
But they are not just *any* trains. Oh no. These are special ones. Special KID trains.
Thomas the Tank Engine.
Yes, that first dragon is Thomas.
OMG that was just about the funniest thing I'd ever seen. Angry Thomas swoops down, breathes fire, perches on buildings and glares. Brilliant!
I'd forgotten how challenging SMAC could be...intellectually the hardest game I've played, because of the sheer range of variability it can have. It's a mostly 2D game.
There's a randomly generated world map; there's a pre-created standard one (actually, probably several). Maps can have several sizes, S/M/L/XL. They can have a variable amount of ocean surface. Land surface is random. There are seven skill/difficulty levels. There's an R&D tech tree, advances are mostly random. Your starting location on the map is random. Ground resources are random. Mobile units that you can create have near-infinite variability of designs that you choose if you wish.
There are seven playable factions; you play one of them, and they're fairly different, different strengths and weaknesses.
If you go with a large map and lots of ocean, the factions are (probably) on islands separated by a fair amount of water. This allows unfettered development with little or no conflict. Then success is based on how fast you expand how far.
Wow. Not easy at all. I had to relearn all the detail. I've only ever played as two factions, green (default) and white. And I've only played two skill levels (default and +1).
I need to try some of the other factions and harder skill levels, now that I've been through it again.
-----
A month ago I received a URL for something on Youtube...it was the opening sequence from Skyrim. You know the one, you're about to get your head chopped off, when all of a sudden, THE DRAGON appears overhead, and you escape in the confusion.
That's the normal situation.
This link was to a machinima of how someone made a game mod to change it. This designer happens to like trains, steam engines in particular, so he's made several mods involving them. There's a spell that rains trains from the sky--do that in town and pretty soon you're Public Enemy #1.
But this startup movie. He's replaced the dragons throughout the entire game with trains. So when you should be hearing the dragon roar, you actually get a train whistle.
But they are not just *any* trains. Oh no. These are special ones. Special KID trains.
Thomas the Tank Engine.
Yes, that first dragon is Thomas.
OMG that was just about the funniest thing I'd ever seen. Angry Thomas swoops down, breathes fire, perches on buildings and glares. Brilliant!
Monday, December 02, 2013
Deus Ex gametime
geez...this was actually going ok for me, except that when i read the walkthrough I discovered I had missed all sorts of stuff. I was enjoying the largely stealth-based gameplay.
And then I got to "the jumping game".
I don't do jumping games. Doesn't work for me--I use wireless mouse and keyboard, they are not properly responsive, and my fingers aren't either.
So we're done with that one.
And then I got to "the jumping game".
I don't do jumping games. Doesn't work for me--I use wireless mouse and keyboard, they are not properly responsive, and my fingers aren't either.
So we're done with that one.
Thursday, November 28, 2013
Dragon Age Origins
So this is a Bioware game, and it sucks like all the other Bioware games I've played, and this will be the last one...
The visual is good, the 3D well done, the building/terrain models good, but as always, it's the game-play that bites almost totally. The game has to be played the way the developers want it played--which is pretty much NOT the way I want to play things.
Sometimes you're solo, sometimes you have a squad. Inevitably, your squadmates will all get killed by opponents because you can't actually manage them quite right, and then you likely will get killed too.
The game is WAY to heavy on their chatty cathies and their cut-scenes (look! we made another mini-movie).
Camera control isn't what I want it to be. Can't much look "up". Can't quite "stealth" enough. Way too heavy on left-hand-keyboard/right-hand-mouse -- which I can't do: I was having RSI trouble on the right hand years ago, so I switched to lefty-mouse, and now my keyboard is different too. Not going back.
Excruciatingly linear, on micro maps.
I did not get very far into it.
Where's the delete button?
The visual is good, the 3D well done, the building/terrain models good, but as always, it's the game-play that bites almost totally. The game has to be played the way the developers want it played--which is pretty much NOT the way I want to play things.
Sometimes you're solo, sometimes you have a squad. Inevitably, your squadmates will all get killed by opponents because you can't actually manage them quite right, and then you likely will get killed too.
The game is WAY to heavy on their chatty cathies and their cut-scenes (look! we made another mini-movie).
Camera control isn't what I want it to be. Can't much look "up". Can't quite "stealth" enough. Way too heavy on left-hand-keyboard/right-hand-mouse -- which I can't do: I was having RSI trouble on the right hand years ago, so I switched to lefty-mouse, and now my keyboard is different too. Not going back.
Excruciatingly linear, on micro maps.
I did not get very far into it.
Where's the delete button?
Friday, November 22, 2013
Steam and their various games
Steam is a fabulous service, the market leader.
But the products are really iffy. I have a number of games that simply do not play on my PC.
Max Payne 1
Max Payne 2
Serious Sam 1 HD
Others are wonky in one way or another, like they sort of work, but have some serious problems and halt for some reason.
Alice 2 (I reach a point where I have to use a custom item, and it simply doesn't do anything)
Batman Arkham City required a mouse that has a different kind of "middle button" than mine.
Supreme Commander 2 has some problem (I forget what was wrong here, but it wouldn't do something).
Do they not do any kind of testing? Or have some minimum testing requirement to impose on game creators to make an attempt at compatibility?
Some work just fabulous
Skyrim
HL 2
Torchlight 1/2
Given the qty of either total or partial failures I've encountered, I only buy games there when they are low-priced, under $20.
And why do so many games insist on installing yet another version of Visual C++ Runtime? or some variant of DirectX? I'm always a little nervous about this.
Wish there was a way to get the broken ones fixed. Or send them an email saying "BROKEN!"
But the products are really iffy. I have a number of games that simply do not play on my PC.
Max Payne 1
Max Payne 2
Serious Sam 1 HD
Others are wonky in one way or another, like they sort of work, but have some serious problems and halt for some reason.
Alice 2 (I reach a point where I have to use a custom item, and it simply doesn't do anything)
Batman Arkham City required a mouse that has a different kind of "middle button" than mine.
Supreme Commander 2 has some problem (I forget what was wrong here, but it wouldn't do something).
Do they not do any kind of testing? Or have some minimum testing requirement to impose on game creators to make an attempt at compatibility?
Some work just fabulous
Skyrim
HL 2
Torchlight 1/2
Given the qty of either total or partial failures I've encountered, I only buy games there when they are low-priced, under $20.
And why do so many games insist on installing yet another version of Visual C++ Runtime? or some variant of DirectX? I'm always a little nervous about this.
Wish there was a way to get the broken ones fixed. Or send them an email saying "BROKEN!"
Batman Arkham City
played some of the PC version of this...it's obviously a console port...visually quite good, possibly the best *looking* game I've played. It's obviously a console port. The controls feel too much exactly like PCGamer Mag always complained about with console ports--not really built for the mouse and keyboard, checkpoint saves
Very much of it is about the keystroke combination sequences that get the Bat Man to do the choreographed motion-animations they probably rotoscoped and re-animated from there. If you can't quite do the special things, it's just a button-masher, which is ultimately kinda boring.
Seems like every notable opponent Batman had is in this, all kinda flat, really depending on you already being very familiar with them. I'm not, of course.
And I have hit a wall. I have to fight the very first opponent who has body armor. It doesn't matter how many times I apply normal hits, it requires me to do a special move THAT MY MOUSE CANNOT DO.
So I'm done playing this one. Which is too bad, because I don't think I really got all that far along.
Moving on to Dragon Age Origins.
Very much of it is about the keystroke combination sequences that get the Bat Man to do the choreographed motion-animations they probably rotoscoped and re-animated from there. If you can't quite do the special things, it's just a button-masher, which is ultimately kinda boring.
Seems like every notable opponent Batman had is in this, all kinda flat, really depending on you already being very familiar with them. I'm not, of course.
And I have hit a wall. I have to fight the very first opponent who has body armor. It doesn't matter how many times I apply normal hits, it requires me to do a special move THAT MY MOUSE CANNOT DO.
So I'm done playing this one. Which is too bad, because I don't think I really got all that far along.
Moving on to Dragon Age Origins.
Some more notes on Parallel FS
OrangeFS looks kinda like what I want, but it has the usual Linux-only aspect.
http://www.orangefs.org/
The lack of real portability bothers me. I do most of my Java Dev work on OSX, 2nd-most on Windows, with Linux a distant third, mostly for concerns about portability. (I do note that that is really the reverse-order list of "able to patch the OS" behavior).
So of course in terms of what I can do for myself, it has to completely be portable across OSes, and not require any sort of "kernel patch" because there's no way I'm going to do that--just not interested.
Orange FS does have similar aspects, but at least in the doc reading I've done, they don't quite have a conceptual theme/analogy.
------
Related, but a little weird: I'm actually thinking about writing some/all of this in Lisp (ok, that'd have some portability issues, but there's no reason a Librarian couldn't be written in Lisp). I haven't done any work in Lisp in years, so that'd be kinda cool...and there are some free Lisp versions that are pretty good these days (I recently re-discovered CLisp for Windows, it's a Cygwin package). Already I can see an issue: I need a build that includes multi-threading, and the basic CLisp does not.
------
Desiderata for my Grid FS:
OS-agnostic
Can make use of any/all machines on the local net (scales)
Doesn't require mount-points for every machine (doesn't scale)
All shared space is available to any machine (scales)
Fault-tolerant about machines coming/going; system contains much self-discovery (scales)
Local apps don't really have to know much about the actual system, they're just going to interact with local files (scales)
Has some amount of redundancy
Doesn't require a bunch of special services
http://www.orangefs.org/
The lack of real portability bothers me. I do most of my Java Dev work on OSX, 2nd-most on Windows, with Linux a distant third, mostly for concerns about portability. (I do note that that is really the reverse-order list of "able to patch the OS" behavior).
So of course in terms of what I can do for myself, it has to completely be portable across OSes, and not require any sort of "kernel patch" because there's no way I'm going to do that--just not interested.
Orange FS does have similar aspects, but at least in the doc reading I've done, they don't quite have a conceptual theme/analogy.
------
Related, but a little weird: I'm actually thinking about writing some/all of this in Lisp (ok, that'd have some portability issues, but there's no reason a Librarian couldn't be written in Lisp). I haven't done any work in Lisp in years, so that'd be kinda cool...and there are some free Lisp versions that are pretty good these days (I recently re-discovered CLisp for Windows, it's a Cygwin package). Already I can see an issue: I need a build that includes multi-threading, and the basic CLisp does not.
------
Desiderata for my Grid FS:
OS-agnostic
Can make use of any/all machines on the local net (scales)
Doesn't require mount-points for every machine (doesn't scale)
All shared space is available to any machine (scales)
Fault-tolerant about machines coming/going; system contains much self-discovery (scales)
Local apps don't really have to know much about the actual system, they're just going to interact with local files (scales)
Has some amount of redundancy
Doesn't require a bunch of special services
Wednesday, August 14, 2013
Amateur software is the bane of my life
an example: I have to parse some CSVs from another tool I didn't write. The folks who wrote it didn't use a standard CSV output generator, and thus failed to generate correct output a disturbing fraction of the time: Specifically: un-escaped embedded quoting characters. And this is in VERSION 2.0 of their software--seems like they never really examined their own output, and that no one else did either, meaning we have already parsed/ingested an unknown amount of data of now questionable validity.
I basically told them "you have to use a standard tool for this--one that YOU didn't write" so as to avoid this kind of thing, one that follows the conventions that pass for a standard for CSVs (which does not have a formal std, but does have well-known syntax).
I recommended three possible choices: Apache Commons CSV (not my fave, it's a little cumbersome), JavaCSV (my fave), and OpenCSV (fewer control options). I think they're going with Apache; I am using JavaCSV.
That is now resolved, but this is a beginner kind of error. Yeesh!
Sunday, June 23, 2013
Distributed File System, part 3
I've thought through a lot of this already, but I have not implemented much. But I have gotten started...
Wish I could put a block diagram thing in here somehow...image seems the only way, but I don't really have any.
So you want to have all the shared files available everywhere, but you sure can't keep copies everywhere, and I discussed the idea of cross-mounts, or buying truly massive storage devices, etc... none of those things are workable, really.
So what I think you do is gather the knowledge of what all the shared files are, catalog them, publish the catalog via a web-service, and then transparently copy things locally when you need to use them, and age them off later (either by a size heuristic, or a time heuristic, or LRU heuristic), making them available locally.
Depending on what's happening across your network, you could wind up with a popular file having copies actually reside a lot of places...for a while. Most files would only have two locations: primary shared, and whatever there is for standard backup,
At the moment I'm thinking of it rather like a library system.You have your own collection of files (books you acquired somewhere). You are willing to share some of them. Others are likewise willing to share some. There's a "library/librarian" service. You can ask the service what all is available (from all those willing to share--the "library" doesn't have its own repository), and you can have a copy of anything listed, until you bump into your local age-off restrictions. Remember how your local physical library works? You can look at the catalog, find something you want, check out a book for 30 days, take it home to be in your personal library, and then return it: i.e., locate a file, copy it locally for temporary use, and then delete it.
If you find yourself having age-off space problems, maybe you buy some bigger bookshelves (i.e., a new and larger disk drive).
This is not a perfect analogy, but works ok for the moment.
So there are some other storage units that could/should participate in this, and they need a proxy of sorts to do so: SAN, NAS--that sort of thing. A NAS device can be just a mountable filesystem, which suggests that perhaps the Librarian needs to take on the management of that, although that doesn't quite fit the analogy the right way: I am thinking of the file-copying as being a lot more like a P2P file-transfer system.
So there's the Librarian service(s), the local shared-publishing service, the P2P file-transferring, and the local storage management. I've written a small part of the Librarian, more of the local shared, I've been looking at file-transfer codes, and merely thought about the storage mgmt. It's all just casual so far, although it's been in the back of my mind for months. Been writing down the use cases, too. I should have a working system in a couple of months, I think.
[Later: ok, I've put less time into it recently, so not til this fall at the earliest]
Wish I could put a block diagram thing in here somehow...image seems the only way, but I don't really have any.
So you want to have all the shared files available everywhere, but you sure can't keep copies everywhere, and I discussed the idea of cross-mounts, or buying truly massive storage devices, etc... none of those things are workable, really.
So what I think you do is gather the knowledge of what all the shared files are, catalog them, publish the catalog via a web-service, and then transparently copy things locally when you need to use them, and age them off later (either by a size heuristic, or a time heuristic, or LRU heuristic), making them available locally.
Depending on what's happening across your network, you could wind up with a popular file having copies actually reside a lot of places...for a while. Most files would only have two locations: primary shared, and whatever there is for standard backup,
At the moment I'm thinking of it rather like a library system.You have your own collection of files (books you acquired somewhere). You are willing to share some of them. Others are likewise willing to share some. There's a "library/librarian" service. You can ask the service what all is available (from all those willing to share--the "library" doesn't have its own repository), and you can have a copy of anything listed, until you bump into your local age-off restrictions. Remember how your local physical library works? You can look at the catalog, find something you want, check out a book for 30 days, take it home to be in your personal library, and then return it: i.e., locate a file, copy it locally for temporary use, and then delete it.
If you find yourself having age-off space problems, maybe you buy some bigger bookshelves (i.e., a new and larger disk drive).
This is not a perfect analogy, but works ok for the moment.
So there are some other storage units that could/should participate in this, and they need a proxy of sorts to do so: SAN, NAS--that sort of thing. A NAS device can be just a mountable filesystem, which suggests that perhaps the Librarian needs to take on the management of that, although that doesn't quite fit the analogy the right way: I am thinking of the file-copying as being a lot more like a P2P file-transfer system.
So there's the Librarian service(s), the local shared-publishing service, the P2P file-transferring, and the local storage management. I've written a small part of the Librarian, more of the local shared, I've been looking at file-transfer codes, and merely thought about the storage mgmt. It's all just casual so far, although it's been in the back of my mind for months. Been writing down the use cases, too. I should have a working system in a couple of months, I think.
[Later: ok, I've put less time into it recently, so not til this fall at the earliest]
Friday, June 21, 2013
The annual V-day M/F relationships writings...
you see online...
[this blog post was started in 2011 and then forgotten for a while]
There were several interesting ones this year [2011]. The first was from a Mormon woman, let's say late 20s, in a big city (possibly NYC, but I don't recall). She was lamenting the usual "can't find a man" situation. So of course she had some requirements that weren't being met: same religion, no pre-marital sex. IIRC, she was unhappy that guys would not stick around long; not like she wasn't a good catch: good education, good job. Eventually one of them made it clear to her: "You left nothing for us to be/do in your life" (Mormons being still a bit more traditional per historical attitudes.) In other words: you are sufficiently independent that we have no self-perceived value in the relationship--how can we be a "provider" when you don't need that?
A male comment on an entirely different story I read some weeks later put it better: "We need to be needed." When we aren't, well, it's time to leave.
So around V-day there was a story by a woman in NYC who basically said to other women: "Can't find a man? It's you, not them." It went right to the heart of things: what you say you want and what you do aren't the same. Of course there was a firestorm of comments in response. Many were a little off-target ("Why the assumption that every woman needs a man?" -- you have to wonder why those folks even read the story to begin with, and then complained, they aren't the target audience). The author was herself having this trouble, thinks NYC demographics are part of the problem (apparently there are noticeably more single women than men there), but blames herself for essentially pursuing the excitement factor and variety rather than something else.
HU sez: "don't bitch about there being no good men--if you haven't found one then that isn't what you want."
[this blog post was started in 2011 and then forgotten for a while]
There were several interesting ones this year [2011]. The first was from a Mormon woman, let's say late 20s, in a big city (possibly NYC, but I don't recall). She was lamenting the usual "can't find a man" situation. So of course she had some requirements that weren't being met: same religion, no pre-marital sex. IIRC, she was unhappy that guys would not stick around long; not like she wasn't a good catch: good education, good job. Eventually one of them made it clear to her: "You left nothing for us to be/do in your life" (Mormons being still a bit more traditional per historical attitudes.) In other words: you are sufficiently independent that we have no self-perceived value in the relationship--how can we be a "provider" when you don't need that?
A male comment on an entirely different story I read some weeks later put it better: "We need to be needed." When we aren't, well, it's time to leave.
So around V-day there was a story by a woman in NYC who basically said to other women: "Can't find a man? It's you, not them." It went right to the heart of things: what you say you want and what you do aren't the same. Of course there was a firestorm of comments in response. Many were a little off-target ("Why the assumption that every woman needs a man?" -- you have to wonder why those folks even read the story to begin with, and then complained, they aren't the target audience). The author was herself having this trouble, thinks NYC demographics are part of the problem (apparently there are noticeably more single women than men there), but blames herself for essentially pursuing the excitement factor and variety rather than something else.
HU sez: "don't bitch about there being no good men--if you haven't found one then that isn't what you want."
Game Philosophy
What causes a game to be successful? Do you need an Advanced Degree (tm) to figure it out?
To what extent is a game's success based on:
What do I mean here?
Visuals/graphics: the very best-looking games these days are things like Skyrim, CoD, etc. Fabulous 3D world to wander through. I love Skyrim (altho I think I like Oblivion better, for reasons of greater variety); visually stunning. But other games, less good, have decent graphics, too, and some interesting games have fairly limited graphics. I've replayed Total Annihilation recently (from GOG, despite my having the original install disk), and heck, that's only just barely 3D at all, it's 8-bit color, etc, and yet that doesn't matter in the end--it can still be quite difficult.
Story: Skyrim etc have pretty good stories in them. There's a main plot, and some relevant/valuable major sub-plots, and lots of little tiny things. This all works great. In fact, those little side projects work so well, I haven't even started the main plot yet, and that's after several hundred hours of game time.
Action: Quake 1-3, Unreal Tournament, etc, are all about the action. The 3D-ness of the maps is interesting, but not critical. There's no story whatsoever. For me, this makes for limited interest. Replayability is all about improving your twitch skill. I enjoy the speed and action, but the only real interesting thing about replayability is that you can do it in relatively tiny increments, like 5-10 mins.
Explorability: Half-Life 2 is a great game, but it gets a zero on this scale. It's very linear. Too linear. Dungeon Siege 1 is equally linear (well, nearly so), but you have a lot of leeway in how you play your character/team. Skyrim etc are anything BUT linear--you don't EVER have the play the main story. I like this aspect--I really don't like being locked into playing a game only one possible way, being locked into a developer's limitations--they might as well do machinima of it for you. I'm not suggesting that linearity = ease of play, it means no opportunity to meander around and look at things.
Good AI: This doesn't even apply to a wide range of games. Team Fortress 2, UT04, Quake3, etc. The AI is other humans. Alpha Centauri, otoh, is mostly AI, and can be really hard to take on.
Other: not sure what I think this is, but maybe it's something like you can find in MMO games, where you can participate without exactly being a quest player, like by being a "crafter". This doesn't interest me. I actually felt more distracted by this whole routine. DLC is a new aspect.
Think back a bit on all your games...Pong, 40 years ago, was the absolute minimalist graphics game, but was not at all easy--it was action-only, no AI, playable in tiny increments; Tank was much the same, only very slightly more complex. This was the era when graphics were super-limited. Think of other games where this is some better, but still the game has to be dominated by something else--while better-looking, Diablo is a little less about action, it seems more about the process of managing loot and such like. There's some story of sorts, but I wasn't really keeping track of that too well--despite the maps being mostly unique each play-through, it's still fairly linear.
So where is the trade-off sweet-spot? I'm sure there's a range. Could we describe it, put some bounds on it? Maybe more by example than by measurement. Reason I ask: I have developed a game or two in the distant past (known as the 70s), and have contemplated making one again, but I find myself debating what flavor I would create. Certainly it would avoid things I dislike, like the repair/crafting stuff. I'd want auto-generated maps to maximize replayability. I'd want to have some reasonable amount of action, but not where it devolves into a twitch game. I'd want some reasonable amount of story; I think I'm more story-driven than most folks. My son is more action-oriented, it seems, he can play TF2 for hours/days; he has, however, played Oblivion et al about as much as I have, HL2 more, Mass Effect, Fallout3/FNV more...he does have more time right now, but that won't last.
How much work goes into making a good story? Is it really all that much? If it's not, you should be able to take one of the free "game engines" and make a game. How difficult is it? How do you make it a story you can actually participate in, as opposed to just following a script? Think of making a game from a movie: seems over-constrained.
It seems to me that good story is what really makes a game--for the kind where there even IS a story. Think about it--I think we tolerate less-than-photorealistic visuals for a better story.
So how hard is it to make a really good story? Do you need more than one? Is it even possible to have more than one? They'd mostly have to be disjoint. Perhaps retirement is the time for me to tackle creating a better story for a game. The problem with that is that it is probably going to still feel too linear. If you allow much variability it's going to become very hard to manage reaching a pre-defined endgame conclusion. My goal would probably be to aim for a much less predictable outcome: create a starting point, play rules, and run it more like a simulation, and watch to see what happens.
I need to re-experiment with some AI activities. Can I make something that is largely emergent-behavior and interesting?
To what extent is a game's success based on:
- Visuals/graphics
- Story
- Action
- Explorability
- Good AI
- Other
What do I mean here?
Visuals/graphics: the very best-looking games these days are things like Skyrim, CoD, etc. Fabulous 3D world to wander through. I love Skyrim (altho I think I like Oblivion better, for reasons of greater variety); visually stunning. But other games, less good, have decent graphics, too, and some interesting games have fairly limited graphics. I've replayed Total Annihilation recently (from GOG, despite my having the original install disk), and heck, that's only just barely 3D at all, it's 8-bit color, etc, and yet that doesn't matter in the end--it can still be quite difficult.
Story: Skyrim etc have pretty good stories in them. There's a main plot, and some relevant/valuable major sub-plots, and lots of little tiny things. This all works great. In fact, those little side projects work so well, I haven't even started the main plot yet, and that's after several hundred hours of game time.
Action: Quake 1-3, Unreal Tournament, etc, are all about the action. The 3D-ness of the maps is interesting, but not critical. There's no story whatsoever. For me, this makes for limited interest. Replayability is all about improving your twitch skill. I enjoy the speed and action, but the only real interesting thing about replayability is that you can do it in relatively tiny increments, like 5-10 mins.
Explorability: Half-Life 2 is a great game, but it gets a zero on this scale. It's very linear. Too linear. Dungeon Siege 1 is equally linear (well, nearly so), but you have a lot of leeway in how you play your character/team. Skyrim etc are anything BUT linear--you don't EVER have the play the main story. I like this aspect--I really don't like being locked into playing a game only one possible way, being locked into a developer's limitations--they might as well do machinima of it for you. I'm not suggesting that linearity = ease of play, it means no opportunity to meander around and look at things.
Good AI: This doesn't even apply to a wide range of games. Team Fortress 2, UT04, Quake3, etc. The AI is other humans. Alpha Centauri, otoh, is mostly AI, and can be really hard to take on.
Other: not sure what I think this is, but maybe it's something like you can find in MMO games, where you can participate without exactly being a quest player, like by being a "crafter". This doesn't interest me. I actually felt more distracted by this whole routine. DLC is a new aspect.
Think back a bit on all your games...Pong, 40 years ago, was the absolute minimalist graphics game, but was not at all easy--it was action-only, no AI, playable in tiny increments; Tank was much the same, only very slightly more complex. This was the era when graphics were super-limited. Think of other games where this is some better, but still the game has to be dominated by something else--while better-looking, Diablo is a little less about action, it seems more about the process of managing loot and such like. There's some story of sorts, but I wasn't really keeping track of that too well--despite the maps being mostly unique each play-through, it's still fairly linear.
So where is the trade-off sweet-spot? I'm sure there's a range. Could we describe it, put some bounds on it? Maybe more by example than by measurement. Reason I ask: I have developed a game or two in the distant past (known as the 70s), and have contemplated making one again, but I find myself debating what flavor I would create. Certainly it would avoid things I dislike, like the repair/crafting stuff. I'd want auto-generated maps to maximize replayability. I'd want to have some reasonable amount of action, but not where it devolves into a twitch game. I'd want some reasonable amount of story; I think I'm more story-driven than most folks. My son is more action-oriented, it seems, he can play TF2 for hours/days; he has, however, played Oblivion et al about as much as I have, HL2 more, Mass Effect, Fallout3/FNV more...he does have more time right now, but that won't last.
How much work goes into making a good story? Is it really all that much? If it's not, you should be able to take one of the free "game engines" and make a game. How difficult is it? How do you make it a story you can actually participate in, as opposed to just following a script? Think of making a game from a movie: seems over-constrained.
It seems to me that good story is what really makes a game--for the kind where there even IS a story. Think about it--I think we tolerate less-than-photorealistic visuals for a better story.
So how hard is it to make a really good story? Do you need more than one? Is it even possible to have more than one? They'd mostly have to be disjoint. Perhaps retirement is the time for me to tackle creating a better story for a game. The problem with that is that it is probably going to still feel too linear. If you allow much variability it's going to become very hard to manage reaching a pre-defined endgame conclusion. My goal would probably be to aim for a much less predictable outcome: create a starting point, play rules, and run it more like a simulation, and watch to see what happens.
I need to re-experiment with some AI activities. Can I make something that is largely emergent-behavior and interesting?
Wednesday, June 19, 2013
Distributed File System, part 2
In April, I had a training class called "Intro to Big Data", from Learning Tree. It's really aimed at your getting into Hadoop, but prelim topics were covered first with separate tools. Nice course, really. LT is clearly good at this kind of thing (was my first/only LT course), unlike some other "training" I've had in the last year.
So what sparked my thinking again on DFS/VFS was the segment about Distributed Hash Table. That might work as the lookup mechanism I need to have server as the complete distributed file table.
Making a distributed database is not easy, even the big guys have trouble with this, and overall performance is not all that great. My fave SQL database, H2, is not distrib.
I do not, as yet, know anything about what sort of performance I need. *I* probably don't need all that much, but running my Grid Engine would need more.
Suppose I take a DHT tool (Apache Cassandra is one possibility) have it store this:
filename, directory path, host
where filename is the access key, and maybe host/path is stored as a URL.
filename, URL
If the URL is good, I could pass it to the Grid Engine as is, and let relevant/interested process(es) use it directly to open a file stream. That could work; it could mean having a lot of file streams/handles open at any one time. (The GE typically wouldn't have more than 100 at a time per machine, probably. Well, maybe 200.) So depending on file size, maybe that's too much network traffic; if nothing else, it's not going to scale well.
Maybe I should be using the file-content MD5 as the key? that is at least fixed size (32 chars). That ends up being much more the DHT approach, because you could distribute keys based on the first character of the MD5 (or maybe the first two, if you had a lot of machines).
MD5, URL
So what am I doing with these things? Suppose I have what I think a DHT is: a local service which can tell me where a file actually is for a given MD5; that MD5 has come from the Grid Engine. OK, that feels clunky, because I only know MD5s from the GE.
Other tools: HDFS (Hadoop) has several issues: the "ingest problem" (i.e., how do you get all your data into it), internal replication (it wants 3X, although you can set that to just 1X: you lose any redundancy security, but ingest is faster), and block size, since it uses 64MB ??!! That's maybe not so painful if your files are all 2GB video...
Another reason to NOT try to use a huge SAN cluster (you can daisy-chain these things) is that you end up having to have a minimum block size around 4k or 8k. Well, that's fine if your files are mostly big, but what happens when you tend to have a lot of 1K files? That issue argues for VFS which lets you use (for example) a ZIP file as a file system, which probably gets around the minimum block-size problem; I expect that has other performance issues, but wasted space isn't one of them.
So what sparked my thinking again on DFS/VFS was the segment about Distributed Hash Table. That might work as the lookup mechanism I need to have server as the complete distributed file table.
Making a distributed database is not easy, even the big guys have trouble with this, and overall performance is not all that great. My fave SQL database, H2, is not distrib.
I do not, as yet, know anything about what sort of performance I need. *I* probably don't need all that much, but running my Grid Engine would need more.
Suppose I take a DHT tool (Apache Cassandra is one possibility) have it store this:
filename, directory path, host
where filename is the access key, and maybe host/path is stored as a URL.
filename, URL
If the URL is good, I could pass it to the Grid Engine as is, and let relevant/interested process(es) use it directly to open a file stream. That could work; it could mean having a lot of file streams/handles open at any one time. (The GE typically wouldn't have more than 100 at a time per machine, probably. Well, maybe 200.) So depending on file size, maybe that's too much network traffic; if nothing else, it's not going to scale well.
Maybe I should be using the file-content MD5 as the key? that is at least fixed size (32 chars). That ends up being much more the DHT approach, because you could distribute keys based on the first character of the MD5 (or maybe the first two, if you had a lot of machines).
MD5, URL
So what am I doing with these things? Suppose I have what I think a DHT is: a local service which can tell me where a file actually is for a given MD5; that MD5 has come from the Grid Engine. OK, that feels clunky, because I only know MD5s from the GE.
Other tools: HDFS (Hadoop) has several issues: the "ingest problem" (i.e., how do you get all your data into it), internal replication (it wants 3X, although you can set that to just 1X: you lose any redundancy security, but ingest is faster), and block size, since it uses 64MB ??!! That's maybe not so painful if your files are all 2GB video...
Another reason to NOT try to use a huge SAN cluster (you can daisy-chain these things) is that you end up having to have a minimum block size around 4k or 8k. Well, that's fine if your files are mostly big, but what happens when you tend to have a lot of 1K files? That issue argues for VFS which lets you use (for example) a ZIP file as a file system, which probably gets around the minimum block-size problem; I expect that has other performance issues, but wasted space isn't one of them.
Friday, June 14, 2013
Distributed file system, part 1
There's a lot of data around on a lot of computers everywhere...far too much to fit on any one machine, or even on some kind of larger storage in any cost-effective manner for us little guys.
At work I have a SAN, 100TB available storage. THAT is a lot of storage; but given what I do there, actually not all that hard to fill up. But that kind of device STILL does not solve the larger problem, nor was it very cost effective--I could replace the drives, from 2TB to 3TB, but that would only be a 50% increase...suppose I need a 10X increase? 100X? More?
2TB drives aren't very expensive any more (you know, it seems almost absurd to even be able to say that, given that my first computer had a 20 MB drive in it), and it's not hard to find dirt-cheap machines around, used or even free. Regrettably they are seldom small, and therefore tend to be a little power hungry...not a prob for a data center kinda place, but uncomfortable for me at home.
Suppose I decided I had a problem to work where 30TB looked like the right capacity...and let's say that means 10 machines @ 3TB each...
I've written a heterogeneous distributed OS-agnostic Grid Engine. Perfect for doing data processing on a 10-node cluster. But this really works best when all the nodes are using a shared/common file system. THAT works best with a SAN and a Blade Server, like at work. Well, the blade server part isn't really very expensive ($3k will buy a decent used one that is full, and pleasantly fast--look on EBay for IBM HS21 systems). But getting a SAN on there--not going to happen. OK, I could perhaps put some high-cap 2.5" drives in the blades, etc, but that doesn't solve the resulting problem, which is still how do they share data with each other?
Well, on a limited basis you can make file shares and cross-mount all the shares across all the machines--but that doesn't scale all that far, and those shares all become a nightmare--and they STILL aren't a shared common file system.
So really the problem I have is how to make a shared common file system across a bunch of machines? I need it to be heterogeneous, since I run Mac/Win/Linux machines, and am considering other things like Gumstix.
There are homogeneous file systems around...several, it turns out, although they are mostly Linux-only (FUSE, Lustre/Gluster, etc), which doesn't help me. OK, I could just buy the cheap hardware, and install Linux everywhere, but what happens when I have a windows-only software tool to run?
I've been hunting for an OS-agnostic tool, it's not really clear whether there is such a thing. OpenAFS (i.e, Andrew File System) might do it, which would be perhaps the ideal solution. I haven't tried this yet. Pretty much everything I've read about doesn't meet my requirements, heterogeneous being the first fail point. At work I'm using StorNext with the SAN, but I can't afford that on my own.
So I think I have to solve this myself. What I kinda think I want is a BYOD approach where you'd have to run some agents to join, but you'd have access to everything shared on the network without having to cross mount a zillion things that you can't even find out about casually.
What you would NOT have is something that shows up in Finder/Windows-Explorer. I can probably figure out how to finagle that too, altho I don't consider that a critical requirement. I expect that OpenAFS has that figured out.
Is it going to take an Advanced Degree(tm) to figure this out? It's not an easy problem.
At work I have a SAN, 100TB available storage. THAT is a lot of storage; but given what I do there, actually not all that hard to fill up. But that kind of device STILL does not solve the larger problem, nor was it very cost effective--I could replace the drives, from 2TB to 3TB, but that would only be a 50% increase...suppose I need a 10X increase? 100X? More?
2TB drives aren't very expensive any more (you know, it seems almost absurd to even be able to say that, given that my first computer had a 20 MB drive in it), and it's not hard to find dirt-cheap machines around, used or even free. Regrettably they are seldom small, and therefore tend to be a little power hungry...not a prob for a data center kinda place, but uncomfortable for me at home.
Suppose I decided I had a problem to work where 30TB looked like the right capacity...and let's say that means 10 machines @ 3TB each...
I've written a heterogeneous distributed OS-agnostic Grid Engine. Perfect for doing data processing on a 10-node cluster. But this really works best when all the nodes are using a shared/common file system. THAT works best with a SAN and a Blade Server, like at work. Well, the blade server part isn't really very expensive ($3k will buy a decent used one that is full, and pleasantly fast--look on EBay for IBM HS21 systems). But getting a SAN on there--not going to happen. OK, I could perhaps put some high-cap 2.5" drives in the blades, etc, but that doesn't solve the resulting problem, which is still how do they share data with each other?
Well, on a limited basis you can make file shares and cross-mount all the shares across all the machines--but that doesn't scale all that far, and those shares all become a nightmare--and they STILL aren't a shared common file system.
So really the problem I have is how to make a shared common file system across a bunch of machines? I need it to be heterogeneous, since I run Mac/Win/Linux machines, and am considering other things like Gumstix.
There are homogeneous file systems around...several, it turns out, although they are mostly Linux-only (FUSE, Lustre/Gluster, etc), which doesn't help me. OK, I could just buy the cheap hardware, and install Linux everywhere, but what happens when I have a windows-only software tool to run?
I've been hunting for an OS-agnostic tool, it's not really clear whether there is such a thing. OpenAFS (i.e, Andrew File System) might do it, which would be perhaps the ideal solution. I haven't tried this yet. Pretty much everything I've read about doesn't meet my requirements, heterogeneous being the first fail point. At work I'm using StorNext with the SAN, but I can't afford that on my own.
So I think I have to solve this myself. What I kinda think I want is a BYOD approach where you'd have to run some agents to join, but you'd have access to everything shared on the network without having to cross mount a zillion things that you can't even find out about casually.
What you would NOT have is something that shows up in Finder/Windows-Explorer. I can probably figure out how to finagle that too, altho I don't consider that a critical requirement. I expect that OpenAFS has that figured out.
Is it going to take an Advanced Degree(tm) to figure this out? It's not an easy problem.
Sunday, May 05, 2013
Young female guitar players
been You-tubin some the last few days...cause a friend pointed me at Joe Bonamassa on YouTube. OK, so I don't know who he is...he's good, but obscure. His better work seems to be when he's playing with someone else more famous and with better tunes. (Later: just found out that YT app is available on iphone from Google--yay! After Apple punted theirs)
and then I happened on Orianthi. and then Desiree Bassett.
Wow.
Neither of them is really old enough to play the blues, in terms of the negative life experiences that hammer your soul the right way. So their original "songs" aren't that great...so you're really there to listen to the guitar work. And that part is phenomenal, most of the time (example where it's not: Desiree plays Jeff Beck's "Because We've Ended as Lovers", pretty much note perfect, as though she had learned it from sheet music--problem is, that's supposed to be a really mournful, melancholy tune, and she plays it with way to sharp an edge, too upbeat)
I wanted to be that kinda good on guitar, but I'm not.
There are a few others...Ariel...Juliette Valduriez.
----------
Been therefore digging on a few other YouTube things...Clapton's Crossroads festivals, which I hadn't heard of before (with DVDs now on order)...Concert For George (which seems not to be available in full on DVD, but the whole thing is on YouTube? [although blocked])
Later: concert DVDs arrived, that is some nice stuff. Now to rip the audio and import into ITunes.
and then I happened on Orianthi. and then Desiree Bassett.
Wow.
Neither of them is really old enough to play the blues, in terms of the negative life experiences that hammer your soul the right way. So their original "songs" aren't that great...so you're really there to listen to the guitar work. And that part is phenomenal, most of the time (example where it's not: Desiree plays Jeff Beck's "Because We've Ended as Lovers", pretty much note perfect, as though she had learned it from sheet music--problem is, that's supposed to be a really mournful, melancholy tune, and she plays it with way to sharp an edge, too upbeat)
I wanted to be that kinda good on guitar, but I'm not.
There are a few others...Ariel...Juliette Valduriez.
----------
Been therefore digging on a few other YouTube things...Clapton's Crossroads festivals, which I hadn't heard of before (with DVDs now on order)...Concert For George (which seems not to be available in full on DVD, but the whole thing is on YouTube? [although blocked])
Later: concert DVDs arrived, that is some nice stuff. Now to rip the audio and import into ITunes.
Advanced Software, leading to PDVFS
I generally work on somewhat exotic software projects. Cutting, if not bleeding, edge.
Was early in the Semantic Web stuff 2000-06, the AI stuff in the 80s, other oddments like Digital Mapping (starting in the 80s), text processing (starting mid-90s), I wrote one of the very first GUI builders (late 80s). A health-care R&D effort in the early 90s would still be cutting edge today.
My latest bit of exotic is a Grid Engine. Granted, not anything new, other than mine is OS-agnostic. You can readily find the other grid engines, but they are not really agnostic. Mine runs Windows (XP/7), Linux (probably any flavor) and OSX (at least 10.6+). The whole thing is of course written in Java, which is why it's agnostic. It should run anywhere a Java 1.6 JVM runs properly (possibly including JME, I haven't a way to test there--it would depend on the lightweight thread support).
I'm now processing a lot bigger datasets than I used to, thus the Grid Engine, in order to distribute processing adequately. I have, so far, run it on two systems: 3 machines with 64 total cores, and 12 machines with 48 total cores. It's designed to run on A LOT of machines, but I'm pretty sure that there are undiscovered scale-up problems along the way. There's no imposed maximum.
Because the datasets are now bigger, I have to think about additional problems. In particular, where does that data go? Everything is fine as long as the dataset is under 2 TB, because that fits a single disk just fine, but then you have the issue of how many clients have to be served by that disk, and therefore how much punishment the disk is taking over time; this is the arrangement I have on the 3/64 machines, with no apparent disk degradation yet. If you use a SAN, you can certainly make a much larger apparent single partition; this is what I have with the 12/48 machines, that's a blade chassis with an attached FC-SAN, with 60/15/15/5/5 TB partitions. You set up the SAN for the partition sizes, and use separate software to manage how the blade units see the SAN; works fine, that's really a lot of space, you CAN daisy-chain another SAN onto it, but that isn't really solving the problem--because I've already burned out two disks in it.
I want/need to distribute data differently, so that I am achieving a more random spread of data over storage devices. I want to work this with the grid engine. I need it to be heterogeneous across random hardware.
So of course Hadoop HDFS sounds like a possible, but there are some reason why not. Hadoop is not oriented around this kind of data, where file sizes range from 100 bytes to 3 Gig. Hadoop wants a 64 MB file-chunk size--I don't have that. I need to use native file systems and disk behavior.
Looking at various experimental file systems, nothing seems to do the right job, or be adequately OS-agnostic. There are several Linus-only possibilities, which are probably closer to what I want other than being Linux-only.
Initially I thought I wanted real mounted file-systems. AFS seemed the likeliest solution, but I think that has some problems likely. I don't know what, specifically, except that I wonder what it means to be writing files out--where are they? It looks like a unified file-system, DOES appear OS-agnostic, but...I don't know.
So I'm now thinking about something that isn't actually a file-system, but a P2P-FS-like thing. I need some not-quite-normal capabilities. And I ultimately want it to run on anything that has file storage (or fronts for it, like a NAS). Going to be interesting working this...
Was early in the Semantic Web stuff 2000-06, the AI stuff in the 80s, other oddments like Digital Mapping (starting in the 80s), text processing (starting mid-90s), I wrote one of the very first GUI builders (late 80s). A health-care R&D effort in the early 90s would still be cutting edge today.
My latest bit of exotic is a Grid Engine. Granted, not anything new, other than mine is OS-agnostic. You can readily find the other grid engines, but they are not really agnostic. Mine runs Windows (XP/7), Linux (probably any flavor) and OSX (at least 10.6+). The whole thing is of course written in Java, which is why it's agnostic. It should run anywhere a Java 1.6 JVM runs properly (possibly including JME, I haven't a way to test there--it would depend on the lightweight thread support).
I'm now processing a lot bigger datasets than I used to, thus the Grid Engine, in order to distribute processing adequately. I have, so far, run it on two systems: 3 machines with 64 total cores, and 12 machines with 48 total cores. It's designed to run on A LOT of machines, but I'm pretty sure that there are undiscovered scale-up problems along the way. There's no imposed maximum.
Because the datasets are now bigger, I have to think about additional problems. In particular, where does that data go? Everything is fine as long as the dataset is under 2 TB, because that fits a single disk just fine, but then you have the issue of how many clients have to be served by that disk, and therefore how much punishment the disk is taking over time; this is the arrangement I have on the 3/64 machines, with no apparent disk degradation yet. If you use a SAN, you can certainly make a much larger apparent single partition; this is what I have with the 12/48 machines, that's a blade chassis with an attached FC-SAN, with 60/15/15/5/5 TB partitions. You set up the SAN for the partition sizes, and use separate software to manage how the blade units see the SAN; works fine, that's really a lot of space, you CAN daisy-chain another SAN onto it, but that isn't really solving the problem--because I've already burned out two disks in it.
I want/need to distribute data differently, so that I am achieving a more random spread of data over storage devices. I want to work this with the grid engine. I need it to be heterogeneous across random hardware.
So of course Hadoop HDFS sounds like a possible, but there are some reason why not. Hadoop is not oriented around this kind of data, where file sizes range from 100 bytes to 3 Gig. Hadoop wants a 64 MB file-chunk size--I don't have that. I need to use native file systems and disk behavior.
Looking at various experimental file systems, nothing seems to do the right job, or be adequately OS-agnostic. There are several Linus-only possibilities, which are probably closer to what I want other than being Linux-only.
Initially I thought I wanted real mounted file-systems. AFS seemed the likeliest solution, but I think that has some problems likely. I don't know what, specifically, except that I wonder what it means to be writing files out--where are they? It looks like a unified file-system, DOES appear OS-agnostic, but...I don't know.
So I'm now thinking about something that isn't actually a file-system, but a P2P-FS-like thing. I need some not-quite-normal capabilities. And I ultimately want it to run on anything that has file storage (or fronts for it, like a NAS). Going to be interesting working this...
Saturday, May 04, 2013
The cars again
Right after xmas I (for some reason I forget) discovered that there were meeces in the garage. I should have put the poison out immediately, but I didn't. They ended up getting inside my lovely XKE and doing some chewing on things. For meeces, and a car that age, that means seat cushion insides, and cloth-covered wiring. So now I have some flaky wiring behavior: sometimes the dashboard instruments cut out. Grr.
I hate meeces to pieces.
I hate meeces to pieces.
Subscribe to:
Posts (Atom)