Data Center: Plug-in cards for HP DL360/380

 I don't know how many cards there are that could go in, these are just the ones that I have and have used.

There are two kinds of sockets in the back that I have experience with. Most are PCI, one is called Flexible LOM. Both 360 and 380 servers have the LOM socket, it actually direct-connects to the mobo. The PCI sockets are on riser cards; a 360 can 2 or 3 PCI cards and one LOM, a 380 can have 3, 6, or perhaps 8, and one LOM.

USB expansion: this PCI card has four USB sockets; one is internal only, maybe for an internal thumb drive? I don't have a server where I need extra USBs. Keyboard and mouse is all, except when I have to install OS from a thumb drive. I am now using PXE-boot so generally I don't even need the thumb, and I would go to the Terminal UI for that situation. Maybe an external DVD drive?

4X GbE network card: four standard gigabit ethernet ports. They work exactly like you think. This is a Flexible LOM card.

2X 10GbE network card: copper RJ-45 network ethernet ports. These too work like you expect except they are faster. This is a Flexible LOM card. the 10GbE you are going to want to use the special cables.

2x 10GbE fiber SFP network ethernet ports. This is a Flexible LOM card. Takes standard SFPs.

I expect there's a fiber-channel adapter LOM card, as well as PCI cards. Maybe Infiniband?

The machines only have one LOM socket. There are other PCI sockets. In a DL380 you could have 6 PCI sockets. Any of those can have a network card, but not one of the LOM cards. I have several 380s with six PCI slots.

The presence or absence of these cards has nothing to do with whether they are configured for action, that's a whole different thing.

I still don't know whether adding the fiber LOM card "turns off" the RJ45 ports. It has seemed to in the past, but that might have just been me not knowing what was supposed to happen. There was a time, for me, about a year ago when the ProxMox 8.0 installer didn't have proper device drivers for fiber, and I was trying to install on a machine that had no copper ethernet, just the Fiber LOM card. PM installed ok, but didn't pick up time, or DHCP, or any active networking. That would of course wanted a live network connection, which it didn't have, so it couldn't update itself to use fiber because it needed the fiber already working in order to do the update. StOOpid. A few months later a newer installer of PM had the fiber driver, so all was good.

There's a new thing I need to about operating on disk drives, requires a new flavor of SAS interface card similar to the RAID card, but I'm going to use an entirely separate/different computer for this. Assuming success occurs, will blog about that when the time comes.

Data Center Server computers and their disk drives

512 vs 520. 

Learned this one the hard way, didn't understand some details before I spent some money. Fortunately only a little bit of money. One of the sellers tried to warn me, but I didn't know what he was saying, and I didn't understand the situation yet.

There are several features/details about disk drives you want to understand before you buy any and end up with units you cannot use.

I am only addressing the machines I am using, so others may be different. Those machines are:

HP DL360 Gen 6/7/8/9

HP DL380 Gen 7/8/9

All of these machines have internal RAID hardware. Sometimes that hardware is a removable daughter board, sometimes its directly on the mobo; I have both. I have 410, 420, 440 model/version numbers. 

Sometimes the RAID software interface is an old-school text interface, specifically Gen 6 and 7, with a mousable GUI for Gen 8/9 (and presumably 10/11).

Your first criteria of concern is the physical size, either 2.5 inch or 3.5 inch, and the quantity that the machine can mount.

For the machines that will take 3.5 inch: comes in both 1U and 2U. You will need matching caddy trays that are 3.5 inch and match the machine. Again, Gen 6/7 are different from Gen 8/9+. Finding the right screws is important too, and the screws for a 3.5 inch drive are different from a 2.5 inch. Quantities that can be installed are 4 in a 1U server, and 12 or 15 in a 2U; imagine having 15 28-TB drives in your machine, that's 400 terabytes. (I have seen pictures of another brand 2U server taking 24 drives!)

For the machines that will take 2.5 inch platter or SSD: also comes in both 1U and 2U, quantities vary, from 5 to 8 to 10 to 16 to 24. Platter drives in 2.5 inch are three rotation speeds: 7200, 10K, and 15K. The most common is the 10K. Capacity goes from 36GB to 72GB to 146 to 300, 450, 500, 600, 900, 1.2 TB, and I think there's a 1.8TB unit. I have a few of nearly all those sizes. For a data center you really want the units from 500 to 1.2T. Most all those are 10K, which is good.

The faster a platter spins the faster you can read data from it. 

SSDs are 5-10X faster right out of the box. You really want to use them as much as you can, esp given that in the same physical size you can get up to 8TB per unit. They are a good bit more expensive. EBay will sell you a 500GB 10K 2.5 inch platter drive for $20, a 900GB for $25. These are pretty good deals.

All these details you can read on the device label.

The thing you CAN'T read on the label is what the bytes-per-block/sector number is. There are several possibilities here, 512, 520, and I have one drive that is 4096.

My machines will only take 512. The 520 is an attempt to squeeze a teeny bit more data onto the disk, the 4096 is going to be a bit wasteful but that drive is 10TB so you probably don't ever notice that.

512 vs 520 

Before you click Buy It Now make sure you know the answer to this, else you just bought a brick.

-------------

That said, it is my casual understanding from reading online that it is possible to convert one to the other, but the process sounds a little complicated, and it cannot be done on my machines.

What I have read of that conversion: first off, standard RAID on my machines cannot do it, and will barely even tell me about it. Install one, go to the Gen 8/9 gun-based Smart Storage Administrator, look at details on one of the physical drives, the drawer window on the right will say something like "can't be used for RAID" and will maybe also say "520". So you are going to have to take this drive to another machine that has a daughter card (or PCI) that is itself configured for HBA mode, not RAID, so that software will have more direct access to the drive, then some other software tools (ShredOS apparently) will allow you to turn a 520 into a 512. I have never even gotten close to trying this. One of these days, maybe...

How to build a small Data Center, from scratch

Data Center Article 1.

So I am now trying to build a Data Center that is functional, but not an ugly monster like those in Northern Virginia.

Step One is to reuse an existing building, so I'm not starting a "greenfield build".

There are lots of buildings around that are kinda just basically warehouses, or at least warehouse-like in that they are big empty boxes. Most town of any size have buildings that are empty and are the right minimum size (which I would say is about 8000 sq ft, although 5000 would work too. Anything smaller is probably too expensive to rent.

Step Two is to make sure that building has adequate power. I'm aiming towards about 10 KW per rack, which means NOT doing big AI. If a building has 2-5 megawatts present, or available, that's adequate for a small space.

Step Three is to worry about cooling--are you going to need anything special? For 2-3 MW, probably not. You'll want more than is present in the building unless it was a cold storage building, I would figure on one megawatt of cooling for 3 megawatts of computing.

Step Four is that you need a decision about what software services you are supporting

Step Five is you need a rack design

Step Six is you need a network design

From 2010 through 2014 I was a user of what we now call "cloud services" except that didn't exist with that name at the time. But I needed what is now "rentable cpu", and some reasonable amount of data storage. So what I am trying to build to provide for others is what would have satisfied what I needed but couldn't get 10-15 years ago.

I will address all of those things in this series; some are of course harder than others.

I will also have some sidebar articles about details (such as the ProxMox-on-HP-Gen7).

I am primarily using ProxMox (https://proxmox.com/en/) for Virtual Machines; will probably have some Windows Server 2022 Data Center for folks who need Windows.

I am using NextCloud for casual data storage available to outside users. 

For large internal data storage availability, I am still investigating tools (things like CEPH, for example). I actually started to write my own version of this a few years ago, after having thought about it for a few years around 2011-2012; I ended up not needing it on that project, but that was because I got customer permission to buy a SAN.

Data Center: How to install ProxMox 9 on older servers

Keywords: ProxMox 9, HP DL380 Gen7, installer solved

How to create a small Data Center, Article 2.

How to install ProxMox 9.1 (and likely a few other recent but older versions) onto older rack-mount servers

It took me quite a while, over a year I think, to get to this answer. Not because I spent all that time looking for the answer, but because I didn't really ever find a good enough, or complete, answer.

So here's the problem: ProxMox 9 (8 as well, and apparently some late versions of 7) are different in ways not introduced by the creators (www.proxmox.com), and not detected by the creators, because they aren't test installing on older hardware (specifically HP DL 360/380 Gen 6/7).

What if you need to install on older hardware, for whatever reason? You seem to be SOL. This problem happens on HP DL360 Gen 6 and 7. The problem doesn't happen on Gen 8/9/10/11, because that's newer hardware, newer video hardware specifically, that does in fact include this video mode properly.

But you aren't SOL, although finding out how to get around the problem is non-trivial.

The problem is that while you can get to the initial splash screen and menu, you can't proceed beyond that, because the screen goes black and never recovers.

Why? The newer/est versions of linux installer attempt to set a video mode on the MOBO video electronics that does not exist, the screen goes black and stays that way. The installer doesn't detect a video mode failure, but it can't continue. This problem has nothing to do with a video driver, it is a failure in the hardware, there isn't a solution like "well just update your drivers" like you always have with Windows.

If you read widely enough online, you find a lot of comments to questions where people ask "why is this happening?" but not really much good in the way of answers.

So here's the solution (I did this successfully on five Gen 7 machines, and will be testing on Gen 6 shortly): 

Before you even get to the install, make sure your RAID config is what you want.

Then, when you get that initial splash screen with the four-item list of install options, the first one is "graphical", and on Gen 8/9/10 this works fine, the second one is "Terminal UI" and this is the one you want to use. 

The first menu option, "Graphical", is highlighted, so press down arrow to move the highlight to the second choice, "Terminal UI". Now press "e" (which in this case is short for "edit"). This is a hidden option built into GRUB. Deep in the background there is a file on your installer USB stick that holds a little "grub script" that is being executed to do this install. You can edit this script on-the-fly at this point and add a special codeword in the right place to cause the installed to not set this illegal video mode.

When. you press "e" you get an opportunity to edit the script, the line you want to edit is the one that starts "linuxvm". You want to move the insert cursor to the end of that line, hit space, then enter "nomodeset" which is the magic word that tells linuxvm to NOT set a video mode, but to just use the text interface.

Next press control-x to terminate the editing, that little script will get executed, "nomodeset" will get used, and you will get the "terminal UI" version of all the otherwise graphical (i.e., needs a mouse) entry screens, wanting the same info, and of you go, give the same answers, let it run. It will auto-restart, and if you aren't paying attention to that, you may end up back at the ProxMox installer screen. If that happens, it means you didn't remove the USB stick, so do that and reboot the machine. Now you're good.

You can now integrate this new machine, PM install into you cluster.

Alternate aspects: rather than have to do the "edit" to add "nomodeset" every time, you can modify the original script on your USB stick. This file is "grub.cfg" and you enter nomodeset the same place. This is on line 69 in /boot/grub/grub.cfg

I found this explanation on a Dell website, with no pictures, but apparently their machines have the exact same problem with some older models. I am not a Dell user, so I can't attest to the success of this, but it sounds right. I've lost that URL, sorry. 

You can find plenty of discussion that is just "me too" noise, or partial suggestions that aren't helpful, or non-answers like "have you tried this?", and really no answers anywhere, except for some really peculiar magic keystrokes that sound like unique-to-someone accidents. My solution above will work for all. 

So far I have used this on 5 DL380 Gen 7 and 1 DL360 Gen 6; my other Gen 6 seems to have some other hardware problem that is preventing me even booting the USB Stick; might be that this server is finally dead.

Here's the Dell link where I found the right words:

Dell link for nomodeset   

Reset an Apple Homepod WiFi problem

Just did this today. Found a lot of not-quite-complete info online. I am hoping this helps you resolve this if you're having trouble. 

So here's the starting problem: Homepod has somehow lost internet. I don't know what is going on, it is having serious difficulty playing my local music. I think the actual cause of this is that my ISP-supplied router (a Comcast unit) has flaked out on WIFI. Homepod may have been coalmine canary on this.

Eventually, tho, the iphones were having trouble too. Wired connections are ok, but WIFIs are not. The WIFI broadcast is still alive, but broken--this is why the HP is having trouble. I

 had a second WIFI unit turned on already, was doing some test config for other reasons. So we switched phones to use that. All is good again. That was about a week ago. 

Today it occurred to me that maybe the HomePod had gotten lost in the shuffle, since I hadn't even TRIED to use it for a while. Siri said she had lost internet connection, and "Home" App said something similar. Of course the problem was that HP was configured for the old WIFI, which is still ON but not WORKING. 

So I looked online for how to switch WIFIs on a HomePod. Simple answer: you can't actually do this. Well, in theory you can, but NOT in my situation here. How to fix this? Not as easy as you think.

Solution steps:

1. make sure BlueTooth is turned on on your phone.  I had it off, none of this is going to work without it.
   
2. Tell the "Home" App to forget the HP. Completely forget it. 
3. Do a factory reset on the HP. This means pull the power plug out. Wait 10 seconds. Plug power back in. Touch and hold the top of the HP. Siri is going to tell you it is going to reset and to wait for 3 beeps. 
4. make sure your phone is ON and near the HP, and is logged in on the WIFI SSID you want the HP on; i.e., not the broken CC one I still have. The HP uses your phone, via BlueTooth, to grab settings about logins and whatnot. Essentially the HP is configuring itself to be like your phone. 
1. In actuality that isn't quite what I want, but it's how it works. 
5. You are now configuring the HP from scratch, let it do what it did the first time, etc. Nothing complicated here. 
1. Add the HP in the "Home" App on your phone. 
6. In theory you can play from iphone to HomePod, using "Handoff" but that seems to not work for me. OK, no big deal, it was just a test, not what I actually want. 
7. Any Mac device you have ON THE SAME SUBNET as the HP can see the HP in "Settings->Sounds->Output" and use it as such a device. Because I'm using a new WIFI, it's a new local IP space, and my Mac Pro is upstream and can't see it. 

So I got the old laptop out, got it on the new WIFI ok, it could see the HomePod ok, and itunes can play to it. I want my Mac Pro to play to the HP because that's where all my music is (500GB), so I'm going to have to rejigger this whole network again. I need to do that anyway, and since the CC router has flaked out, I'm going to go ahead and make all that changeover. That will result in a whole new WIFI/wired network setup, new devices, etc, but no big deal that's kinda what I "do for a living" now. 

According to some online reading I did, apparently you can much more easily change the HP WIFI connection IF BOTH ARE ALIVE AND WORKING. Of course that was not my situation, as noted above. 

 I think Siri on HP could do a better job of saying "I'm doing this" or "turn on bluetooth on your phone and hold it close to me". Some pieces of this software suite of iphone and HP and whatever else is running in the background feel unfinished, to me. 

What would ALSO be interesting is if the HP operated via POE, although that's hard to imagine as I expect it wants more raw power than POE is going to provide; there are a lot of flavors of POE, so maybe that's not true. I would prefer that HP be on a wire, tho, but I realize that wireless is the way of things. I'll update this if need be when i do the larger network reconfig.

I'd also like to be able to config the HP via a browser page for the factory reset. I can do this with nearly everything else I have to work with, they have what's called "Management WiFi" which turns on for 15 mins when you power-up the device.

----------

Later: THIS IS WAY NIFTY. Software updates have made it possible to play straight from my iPhone to the HomePod. So I pushed a bunch more music onto the phone.

Am rejiggering my home network

Jazz Music.

Watched Ken Burns Jazz on tv earlier this year. PBS ran it over about 10 weeks.

Interesting stuff. I knew most of the players already, but not all. Had music by a lot of them already.

And now I have about 30 new CDs of them.

The film begins approx 1900 and runs to about 1975. I missed the first episode, so I didn't hear that earliest part.

Weeks later, I couldn't tell you much about the details, but good music.

New Stuff: Basie, Parker, Gillespie, Duke, Miles, Monk...other assortments. Good stuff, should have had some of this before.

COVID and other Vaccinations

I got my shots in March. A bit of an accident at first, but whatever.

Lots of people around the country are not getting the shots, for a variety of reasons that sound mostly stupid to me.

"I don't want to get microchipped" -- as though we can make such a thing small enough to go through a shot needle. Which, btw, are apparently 0.5 millimeters on the inside, bigger than I thought.

"It's made of dead baby parts." That is some sick thinking.

"Happy for the rest of you to get it so I don't have to." That can actually work, and someone I know said this to me. Of course, that person is no engineer/scientist, and could not possibly explain to you what the reality of herd immunity is.

"If God wants to me to die, well, that's it" Maybe he wants you to use your brain. Hard to tell.

But separately from all that, since the "refuseniks" aren't going to become convinced by any outside info...

What would the infection rate have to be, and how severe would the consequences have to be, before they would ignore the garbage they choose to hear/believe, and get it anyway? How quickly would it have to be fatal?

I know some people who have gotten it, and gotten over it, or have died. I don't think I know anyone who got it and clearly has long-term issues; not sure about that one.

But, if C19 was low infection rate but absolutely fatal? High infection but the worst you got was permanent Chronic Fatigue?

My suspicion is that there is a "Big Data" opportunity here that hasn't been seen yet, and awaits discovery. Probably needs more longitudinal data collection, which probably awaits the next few waves of infection.

I understand all the data, the models involved with studying this. Somewhere in my software past I've done some similar modeling and numbers work. You can find plenty of visual models online to show roughly how the infection/immunity process behaves; I'm not making yet another one of them :)

So I don't know if 70% is the magic number here for the herd immunity to be the steady-state situation. There are a lot of variables that are relevant. Example: I wasn't personally too worried about how soon the vaccine would get to me, because I have nearly zero interaction with other people over any time-frame you want to use, so my exposure probability was/is close to zero--that's how my life is these days; a few years ago that was simply not the case (I caught the actual flu once ~10 years ago while on the subway).

If we have a mutated next wave that is worse, what will happen? I suspect we can create and distribute faster, now that we have practice, but still: what will be the refusal rate? How long before the refuseniks die off enough that the remainder get the vaccine?

2020 Word of the year

Excoriate.

you can't get away from this word, in 2020. Seems like hardly a day goes by that I don't see it in some piece of brand-new writing that day.

So you can bet on it taking the prize.

I haven't seen it getting mis-used, yet, but it's getting OVER-used.

Oil prices.

Well, they're scary.

Bad scary. Yesterday (April 20) WTI went to zero. ZERO. Think about that. If the price is zero it means NO ONE is buying.

I used to live in TX, where the "awl bidness" drives the economy. You know folks there are freakin out.

I've found this nifty little javascript widget that shows you what WTI is. There's another for Brent. I'm guessing that the negative means futures contracts being cancelled.


WTI stands for West Texas Intermediate, fwiw.