Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Thursday, May 14, 2026

Data Center -- Installing and Configuring AWS S3 alternatives to be S3 services for internal/external use

My intention is to try out several of these tools. No preference yet. Those I expect to try out: Seaweed FS, RustFS, Garage, LocalStack. (MinIO seems off the table now)

-----

Seaweed FS

 The installation is trivial. Download the tgz file, unzip it, untar the resulting file wherever you want. 


Then there's the startup command:

AWS_ACCESS_KEY_ID=admin \
AWS_SECRET_ACCESS_KEY=secret \
S3_BUCKET=my-bucket \ 
./weed mini -dir=/data

(-dir=/where/the/data/should/go)

This starts the service with some defaults for a tiny space. ("mini")

Make sure that "/data" exists with proper permissions.

I did the install on an HP DL380 Gen 7 with Ubuntu 25 Server. It's not a super-powered machine, but this is just "mini". Will use bigger machine shortly.

This is just for test purposes, you don't want to try to do anything serious/real with "mini". You want to kick the tires. Partly because those are TERRIBLE credentials, and partly because this starts up with a 30GB max space. Kick the tires.

There are multiple web-page interfaces for it:


Obv of course you should use the IP of the server instead of "localhost".

There are some config files to work with on this. The web interface is a lot better at showing you things than it is at letting you change them.

Filer UI is like DropBox in terms of casual file storage in a folder hierarchy. This is the first part I tested. 

Basically I just wanted to see if this did what I was expecting, which it did.

The "Admin/Dashboard" is pretty busy.

To do some REAL work, a REAL system, use "master" instead of "mini": 

./weed master

That's the big one. Also: do this on a machine with a bunch of storage. I am arranging that shortly.

On other machines, you want to also install Seaweed FS, but you will use "volume" instead of "master".

./weed volume -dir="other/machine/file/space/" -max=10 -master="192.168.1.21:9333"

---------------------------------------------------

First step was to configure hardware. See "underlaying steps", below, for details. This was non-trivial, but entirely separate from Seaweed.

Second step was install Proxmox as host OS. This is straightforward, I have done this probably 25 times now. This is using Proxmox 9.2, the latest as of this writing.

Next step was to create the VMs. In Proxmox I created one, converted it to a template, and then created the rest of them. I forgot that I needed to do some work before turning it into a template. Always you do 

sudo apt update

sudo apt upgrade

which I did, but I forgot about setting static IPs on there. You have to fix each VM anyway, but if you do the template first, it's a lot easier. I also should have downloaded seaweed into the original before making the template. Oh well, all set now.

I also set the VMs to auto-start at boot time. I don't want to be doing that by hand ever again. I wish the Proxmox default had that turned on.

Having configured the hardware with big disks, I downloaded Seaweed onto all the VMs. I used "linux-full-amd64.tar.gz". That unpacks into a single executable that is the entirety of Seaweed.


Next I started seaweed.

./weed master

then i looked at the web-page for it, at swhost:9333, which doesn't work. Need to do a total reboot.

Now we are good:


OK. Halt 'weed'. Then 'sudo mv weed /usr/local/bin/'


Now need to do setup for seaweed to auto-start at reboot. It would be better to do this via systemctl, but I didn't get that to work for some reason. I used crontab in the end, with 

@reboot weed master

Reboot a couple times just to be sure, and all is good.

---------------------------------------------------------------

There are some underlying steps you need to make, configuring the hardware and VMs.

The hardware arrangement I have for testing is a single server with 4x16TB HDD. I wondered about how to RAID them, with a 1+0 being the most obvious, but also the most space-wasteful. Other choices like R6 weren't really any better, with only 4 drives. Someone suggest I go with a different approach, where Seaweed itself is providing the redundancy.


One VM runs SW Master. Four VMs run SW Volumes. Configure SW for the amount of redundancy desired, and you have a total a 64 TB

Here's how to configure the host and the VMs. Took a while to get the right info, watch the right videos, deal with the ESL accents.

<picture>

The machine: 1 HP DL 360 G9 4 LFF 128 GB RAM 2x 2640 CPU, + 2x512GB SSD + 2x128GB SSD.


It has a fiber card too, which is not relevant to this writeup:


(Interestingly, this machine includes an internal SSD cage for 4 SSDs. I had two pairs of matching SSDs, so they are /dev/sda (2x512GB)  and /dev/sdb (2x128GB), both configured as RAID 1, where "sda" holds the host OS. I had never seen this cage before.)


The blue are the top two SSDs. There's another pair underneath. This takes special SATA cables to use them, for both power and data. I imagine those come with the cage. The silver cables are data, if you trace them out you can see they plug into the RAID controller. The data cable looks standard, I think I have a couple more of them acquired elsewhere for a different reason; the power cable looks custom to this machine.

The host OS here is ProxMox, so that I can create these VMs all on this one machine. IRL, in the Data Center, I wouldn't do that, each Master and Volume would be separate machines, with Seaweed running in the host OS, not a VM. Probably doing that soon.

But for this investigation...VMs.

Once Proxmox is installed you are going to create 5 VMs. These VMs are going to live inside the 512GB RAID volume, so they need to be no bigger than 80GB when you create them. This is a standard procedure with Prox, not covered here.

Once that is done, using Ubuntu 24 as their OS, need to tell Prox how to make the 16T disks available.

This is done by using the DISK ID rather than "/dev/sdc" names, lest an accident of timing during boot shift those around a little. 

At the command line:

ls -n /dev/disk/by-id/ <return>

and you get this (and probably a lot more)

lrwxrwxrwx 1 0 0  9 May 27 08:02 scsi-3600508b1001c710b4c7493cf3630fb6f -> ../../sdc

lrwxrwxrwx 1 0 0  9 May 27 08:02 scsi-3600508b1001cd17c6f7fea6ce6d097d5 -> ../../sdd

lrwxrwxrwx 1 0 0  9 May 27 08:02 scsi-3600508b1001cc9fc14dafef1c4174c23 -> ../../sde

lrwxrwxrwx 1 0 0  9 May 27 08:02 scsi-3600508b1001c801d4521682fac529211 -> ../../sdf

those are the 4x16T drives. 

/sbin/qm set <vm_number> -scsiN /dev/disk/by-id/scsiNNNNNNNNNNNN

is the generic command. You will have to fill in the blanks of <vm_number>, scsiN, and the disk ID. 

-scsiN specifies the interface used and N is just a made up number, a single-digit in my case.

Ultimately:

/sbin/qm set 102 -scsi2 /dev/disk/by-id/scsi-3600508b1001c710b4c7493cf3630fb6f

/sbin/qm set 103 -scsi3 /dev/disk/by-id/scsi-3600508b1001cd17c6f7fea6ce6d097d5

/sbin/qm set 104 -scsi4 /dev/disk/by-id/scsi-3600508b1001cc9fc14dafef1c4174c23

/sbin/qm set 105 -scsi5 /dev/disk/by-id/scsi-3600508b1001c801d4521682fac529211

This attaches one drive to each of the four VMs (102-105). They will show up as /dev/sdb.

For each VM you will do standard things:

sudo mkfs -t ext4 /dev/sdb

sudo mkdir /mnt/data

sudo chmod a+rwx /mnt/data

sudo mount -t ext4 /dev/sdb /mnt/data

df -k         

will verify that all is good:


and finally edit /etc/fstab for a permanent mount:

/dev/sdb    /mnt/data    ext4    defaults    0    0

sudo mount -a

sudo systemctl daemon-reload

At least as of PM 9.2 the PM GUI doesn't help you do this. It's not hard, per se, but there's clearly a good bit of mechanics.

End result: I have four VMs each with local disk of 60GB and each one has a locally mounted 16TB disk for data. Because those disks are local this is as fast as its going to get.

(if your VMs will run Windows, instead of -scsiN you will use -virtioN instead of -scsiN, and you're going to need a special software driver acquired separately from ProxMox. I wasn't doing this)

-----------------------------------------------

After getting all that setup done, Seaweed volumes running in VMs didn't want to talk to Master, and some further reading led me to the conclusion that VMs are a bad idea here.

So I dropped this whole idea. It turns out you can run more than one Volume Manager along with the Master on the host OS. I reinstalled Ubuntu at the top, and mounted the 16T disks directly. Each one will go with a Volume Mgr, as I was doing in the VMs, but here with better performance, less overhead, and probably correctly working comms. One point in the Seaweed docs it says 8TB maximum space for a Volume, so I may be violating that already. If so, I'll back down to some 3TB spares I have.


Sure enough, right there it says 8000 GB. The disk is 16, so half is wasted? Should I partition that disk into two? This would not have looked good in VMs, there would be twice as many.

And, looking at the Master's web interface:


You can see I fired up 4 Volume servers. I really need to deal with that 8TB limit. And put better CPUs in there; it has E5-2620 Hex-cores, which seems ok for just 4 volume servers, but not enough for 8. Pretty sure I have some 8-core 2640 or something in a box. Maybe I'll spend the $ for some 2698 V3/4s...

FWIW, my test machine has 8 drive slots, 4 for SSDs and 4 for 3.5" HDD. So it's not like I can just put more more smaller drives, 8T or less, and have more volume managers. Yes, I could partition a 16T into two 8T drives, but they aren't operating in parallel on a single drive. 

My impression is that the reason for the 8T limit is because SW is focused on smaller files, and the qty could get crazy with larger Volumes.

On top of that there's some other weirdness going on. Ubuntu HATES having seaweed defined as a "service". That is a disaster, breaks the boot, drops you into recovery mode, giving enough power to turn it off as a service, and then reboot. I don't know what is going on, yet. I can start it just fine from a shell login, seems to run fine, incl the Volume Mgrs, but not as a service enabled at boot time. That can't be right. I couldn't get it to behave by using crontab either. Later: I think I had it autostarting, via servicectl.

Later: It may be that I am doing that Volume Server wrong, and that you want one per machine, it each one handles multiple disks. That has appeared to be the case as I have done further reading. Seaweed is TERRIBLE about documentation with the severe lack of online help, or videos.

But what I think I know now: You want one (or more?) Masters on separate machines, one Volume on each participating machine, one Admin per Master, one Filer (per machine? per Volume?), maybe an S3 if you need that; apparently starting S3 also starts a Filer. And then the WebDAV and other thing, if you ned those. 

When you start Volume Server, you can specify "folders": "-dir=folder,folder,folder" and each folder should be a separate mounted disk, I think, again the 8T limit so I think drives no bigger than that.

Component startup order seems important, not sure why:

Master

Volume

Filer

S3

Admin

Shell

It appears that S3 requires a Filer already running; it seems to hang without one. Admin certainly requires a Master. Shell requires a Master.

----------------------------

Rustfs.

Similar to Seaweed. There's a single executable, same as SW. Where SW is focused on smaller files, RustFS is not. Rustfs is newer, and therefore less stable. There are some good comparisons on YouTube.

It looks similarly easy to start up, although less traditionally organized. One unique feature: it wants to use XFS as the underlying file system. I have no experience with XFS. You can run in 3 modes:

1) one machine one volume

2) one machine multiple volumes

3) many machines many volumes

For my first test I have a couple of under-powered DL360 Gen 6/7 machines that have plenty of RAM but not much disk. They are both configured with a RAID-1 pair of 10K SAS disks to boot from, and the rest of the disks are a RAID-50 group. Ultimately that is then two machines and two volumes. There's no "Master Node" like in Seaweed.

Monday, October 29, 2018

Incorporating H2 database into Eclipse

So I finally did this.

Here's a decent explanation.

https://ibytecode.com/blog/eclipse-dtp-configure-h2-datasource-using-data-source-explorer/

but it goes wrong at the end.

(This is for Eclipse Oxygen; Eclipse Mars can't even do this, on my machine at home; Eclipse Luna doesn't do it either. This is curious, it doesn't feel like a new thing.)


Here's much the same thing over again:

Menubar File->New->Other...

Good god. My pasted-in pictures all got lost. Yeesh. Gotta do them over, saved as files. Yeesh.




Choose "Connection Profile" and "Next"

See this: and click Generic JDBC




Type "H2 JDBC" for the name. (When you try to do this again, you can't use the same name, you'll get told it exists). Click "NEXT", and get a new dialog window.

Click on the strange icon inside the red circle:




See this new dialog:




Click on Generic Driver. Enter H2 JDBC Driver for the name.


If you don't click on Generic, you can't do the next thing:




which is click on "Add JAR". Find your H2 jar file.



Now comes the absurd part. Click on the H2 jar file name. Click on Properties tab button. It's blank. Go back to JAR list. Click the jar file. Click Properties. Still blank. Back to JAR list. Click the H2 file name.

The Edit button will now be active. Click that, the click Cancel in the file dialog. Now click Properties. Yay! props table.


Driver class is org.h2.Driver (capitalized just that way). URL is jdbc:h2:tcp://localhost/wherever-your-db-is. Name is whatever you want as a name. I'm not sure what this is for. Click OK.



enter a password, click "save", click "test connection" to be sure you don't have a typo somewhere.

The tab for "Optional" parameters would be something like mvstore=false; or other jazz.

I don't recommend turning on "connect every time". Regrettably, you'll have to go through this again every time you have a new database, but the end result here is that you can create a .sql file in a project, and connect it to this database, and run sql live right there from in the buffer.



I should have done this years ago...but Eclipse is not really the best of all possible interfaces to H2. Especially since it won't show the result from "SELECT * FROM my_table;", just whether or not it executed error-free.

I have some other tools I wrote for real database interface work, but they don't do text-edit buffer.

--------------------------------

Of course, H2 already has to have that database created...I was trying to get a CSV dumped into a single table to practice some SQL with. Just the load was trickier than I wanted.

How to load a table from CSV into H2:



(Yep, I'm looking at some FCC data.)

H2 doesn't give a very good explanation for CSVREAD to load a DB. Tried it before, and spent an hour today trying to guess what it wanted for syntax. It's actually really simple, simpler than it looks in their doc.

INSERT INTO table(col, col,col...) SELECT col,col,col... FROM csvread("file-location.csv");

There's a simpler way if your table will have exactly the same columns as the CSV, but mine had that extra "ID" column first.

Tuesday, October 09, 2018

Even supposed whiz-kids get it wrong


So I needed to try to export Contacts info in OSX High Sierra to a CSV file. Sounds simple enough, right? If *I* had written the Contacts app, this would be a direct thing off the file menu.

Apple, however, in its infinite wisdom, has completely failed on this. You cannot. Full stop.

You should NOT need an Advanced Degree (tm) to figure this out. But apparently you do.


OK, that said, there is a way, but it's really stupid to have to do it. OK, there might be two. I only tried one.

You have to open Contacts, figure out how to "Select All" (or whatever subset you like), then you open Numbers (yes, Numbers), and click/hold/drag that bunch of contacts over into Numbers (and not just anywhere, you have to land on the actual cells grid. Then you wait a few moments, and poof, the cells are filled with your data. 

From Numbers, you can then do a CSV save. Ye Gods.

That is pointlessly over-complex. CSV is a basic output flavor for contact info--even if you don't look at it in a table (come to think of it, why is THAT not an option? What happens when I need to look at two at once?), it's a table, and. you can always output a table as CSV.

Especially when most things that import tabular data want either a CSV or XLS. Like something I'd need to dump contacts info into.

-----

Looks like Automator might be able to do it. And there maybe is an app in the Store that reads the regular abbu export and makes a CSV. 

Seriously?

Thursday, December 21, 2017

Why is the Intertubes so slow?


Well, of course it is all those others using it.

The other answer is crappy programming on the server side. And on the client side.

Apparently you need an Advanced Degree(™) to do better.

Here's what I mean:

Yes, I'm a Mac User. No, that's not the cause. Except that it is, slightly.



The above image is "Activity Monitor". You can see that the top two *busiest* users of CPU cycles are Safari and Firefox. Everything else is essentially zero, or idle. When a process is ~100%, that means it is saturating a cpu core, and can't do any more.

But those two are busy doing something, and that something makes them fairly unresponsive to what I want them to be doing. (I acknowledge that some of the problem is that while I'm typing this blog, Blogger is busy trying to help me by doing on-the-fly spelling check, which means there's a background process that is sending/receiving content to/from somewhere far away, and that ties things up a little. That help is nice, but I'd rather throttle it back--remember that I am on satellite internet now…)

What are those two up to?

Bad programming. Well, bad websites, and then bad web-broswer. Those numbers are while the browsers are "idle", too. So what are they doing?

Unfortunately, it's hard to find out because the cpu-usage reporting does not go down to the individual-lightweight-thread level, so I can't know which specific open window (URL) is causing the problem. 

But it's a near-certainty that EBAY and AMAZON are the two likeliest culprits, because they are the pages that have the never-ending-animation content. There are probably other URLs that are worse, but ones I don't visit.

Also, look at the real-mem size. Safari is 15GB?  a pig. This is the same problem that Firefox has, too, about lightweight threads that haven't been terminated and not closing file resources.

Which means that we have web-pages that aren't really 100% closed, even if I closed the window/tab. Files aren't close, and the resources that were loaded (images) are still using memory.

Some of the problem is that on this machine I'm on an old OS, Mountain Lion (10.8). Going to bump it up to Capitan (10.10) in ~3 weeks. I don't expect that is going to fix the issue.

But the problem starts with website creation work that insists on putting animation all over. I personally prefer much simpler pages that have no such thing. These animations that force me to see them are offenses of their own. 

You can dodge some of this via things like FlashBlock (Flash use to be THE WORST about this). But that only solves Flash, what about the animated GIFs? No solution I know of.

I agree that newer OS should have some better-behaving Safari in it. And newer machine would help, except that this machine is already ridiculous fast:



So there are problems in browsers: unclosed resources that are dangling, AND mediocre javascript engine. 

Apple is not guilty of these problems in Firefox, but it IS guilty of them in Safari.

All this animation needs to be halted. At least with FlashBlock I could then choose to play something or not.

What I end up doing now is watching URLs that are these criminals, and adding them to etc hosts.

THIS IS NOT TO HARD TO SOLVE, PROGRAMMERS!


[Next day: "Safari Web Content" seg-faulted and killed Safari completely. Geez this is dumb.]

----

If I quit and restart Safari or Firefox, that cpu number goes down to 10% or something like that, then is slowly creeps up again. At 100% the browser is unresponsive, so I'll be ok until I hit that again. But this is ridiculous.

Garbage on the server side causes this, and bad programming on the client/browser side makes it worse.

It's my understanding that javascript is now compiled on the fly, which didn't used to be the case, so it's at least not running slow by being interpreted still. That was never good, just easy.

---

Another issue about why Blogger is problematic: it wants to do this auto-save for you, and as I am on satellite, that fails a bit too often…and then it is stuck in a different way. It might recover in a few minutes, but it is at the same time trying to do the spelling check, so who knows?


Almost enough to make you want to go back to Lynx.

Wednesday, March 30, 2016

another java hacking story, part 3

OK, here's the source.

Note that it does not handle a variety of return codes. I'll work that later as I need them. And I'll come back to this to add things if I remember to.

Completely self-contained in this one file.

Two test cases are included, so you can see that "chunked" return and "normal" return are both handled.

package com.hu.commons;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.MalformedURLException;
import java.net.Socket;
import java.net.URL;
import java.net.UnknownHostException;
import java.util.Enumeration;
import java.util.Properties;

/**
 * AAAAHHHH!
 *
 * This is a scratch rewrite of broken functionality elsewhere.
 *
 * the problem: java.net.HttpURLConnection works just fine when the return is standard, simple, "Content-length: N" where N>0
 *                         BUT IT DOES NOT WORK WITH CHUNKED RESULTS
 *
 * So I tried spring.framework.core.httpclient as an alternative. THAT DIDN'T WORK EITHER. Different, but similar problem about chunked.
 *
 * So I'm making my own replacement, from scratch.
 *
 * This will grow over time to handle more flavors of results. Right now, just 200, 204, 400, 404, 500.
 *
 * using ByteArrayOutputStream as byte accumulator for reading is much simpler.
 *
 * To use: make an instance of NanoHTTPClient. set debug=true if you want verbose output. Call sendMessageGet(url-string) and get a string back.
 *
 * test cases are at httpbin.org and typicode.com

  * for proof about the chunked part, use www.google.com
 *
 * @copyright 2016, Hyde University.
 *
 * feel free to use as you wish, but you may not claim to be the author.
 *
 * find something wrong? oh well.
 *
 */

public class NanoHTTPClient {

    // these really need to be local to instances.
    //probably shouldn't be public either.
    String protocol = "http";
    public String url = null;
    public String host = null;
    public int port = 80; // standard default
    public String method = null; // GET, POST, etc
    public String version = "HTTP/1.1";
    public String serverRev = null; // server's http version
    public String returnCode = null; // 200/400/500/etc
    public String returnMsg = null; // third part of the first line: "OK", "Bad Request" etc
   
    // the in/out header-block properties
    private Properties sendprops = new Properties();
    private Properties receiveprops = new Properties();
   
    boolean debug = false; // aka verbose
   
    // *******************************************************
    // *******************************************************

    public NanoHTTPClient() {}
   
    // *******************************************************
    // *******************************************************

    public void setRequestProperty(String prop, String value) {
        sendprops.setProperty(prop, value);
    }
   
    // *******************************************************
    // *******************************************************

    public String getReeiveProperty(String prop) {
        return receiveprops.getProperty(prop);
    }
   
    // *******************************************************
    // *******************************************************

    // ok, we can't necessarily trust reading characters here and bytes elsewhere
    // so its bytes everywhere
   
    // if your input contains embedded \r\n occurrences, well, this method won't know that, and will assume that's the end.
   
    // it's possible for this to screw up, too, if you input happens to contain a byte sequence
    // that can be interpreted as a valid multi-byte character. Had that happen once.
   
    public String readLine(InputStream is) {
        String result = "";
       
        //ok, we read until we get to the first line break (\r\n)
       
        // BAOS is an accumulator of bytes, you use it like a stream, but you can get a String of the content at any time.
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        try {
            while (true) {
                int b = is.read();
                if (b=='\r') {
                    b=is.read();
                    if (b=='\n') break;
                }
                else
                    baos.write(b);
                }
            } catch (Exception err) {
                err.printStackTrace();
        }
        //ok, make sure we get a clean conversion
        try {result = baos.toString("UTF-8");}
        catch (UnsupportedEncodingException err) {err.printStackTrace();}
       
        if (debug) System.out.println("RDL: " + result);
       
        return result;
    }
   
    // *******************************************************
    // *******************************************************

    // regular old GET call.
   
    // an important thing to understand here, part of the "why" of this code,
    //  is that you can't mix/match a wrapper stream on a base stream and expect
    // reading to behave properly
   
    // this is the problem with java.net.httpclient--there's a base stream in there,
    // httpclient itself reads from that stream, so that if you want to use it,
    // you can't put a wrapper on it (like BufferedReader), because you can't quite expect that the pointer
    // position on that stream is where you think it is after reading the http headers.
   
    // sometimes it isn't; if you mix stream readers, it definitely isn't.
    //  that probably has to do with how the O/S input buffer fits in.
    // imagine that the wrapper clears the i/o buffer when it starts.
    // the wrapper is then at zero, but the underlying stream is, well, *somewhere*
   
    public String callMethodGet (String url) {
        try {
            URL url1 = new URL(url); // this fails on standard reasons, basically the syntax format is wrong.
           
            System.out.println(url1);
           
            //if we happened to be reusing this instance, you want these cleared each time.
            sendprops.clear();
            receiveprops.clear();

            // hold these for later? am I using them yet?
            host = url1.getHost();
            if (url1.getPort()>0) port = url1.getPort(); // default properly
           
            // are these order dependent? nah, apparently not. good.
            sendprops.setProperty("User-Agent", "NanoHTTPClient"); // always use this name, no spoofing that. // or it might need to be that Mozilla/Gecko thing.
            if (sendprops.getProperty("Accept") == null) sendprops.setProperty("Accept", "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,*/*;q=0.5");
            if (sendprops.getProperty("Accept-Language") == null) sendprops.setProperty("Accept-Language", "en-us,ex;q=0.5");
            if (sendprops.getProperty("Accept-Charset") == null) sendprops.setProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
            // this is required, some tiny flavor of security check
            sendprops.setProperty("Host", host + ":" + port);
           
            // ok let's go
            Socket so = new Socket(host, port);
           
            // time to write all the proper stuff to the server
            OutputStream os = so.getOutputStream();
           
            byte[] b = null;
           
            String line = "GET " + url1.getFile() + " " + version + "\r\n"; // getFile includes the query params
            b = line.getBytes();
            os.write(b);
            os.flush(); // don't forget this, you have no idea what buffering is going on behind the scenes.
            if (debug) System.out.println(line);
           
            // do the real work
            // push all the "send" properties across. order not important.
            Enumeration props = (Enumeration)sendprops.propertyNames();
            while (props.hasMoreElements()) {
                String prop = props.nextElement();
                line = prop + ":" + sendprops.getProperty(prop) + "\r\n";
                if (debug) System.out.println(line);
                b = line.getBytes();
                os.write(b);
                os.flush();
            }
            // need an extra blank line
            os.write("\r\n".getBytes());
            os.flush();
           
            // ok that's all for sending to the server
           
            if (debug) System.out.println("-------------");
           
            // *********************************************
           
            // all done. ready to receive.
           
            InputStream is = so.getInputStream();
           
            // easier reading of headers. body is separate handling anyway.
            // no, this won't work right here with multi-byte characters. Plus, you can't mix streams.
            // BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
            // I deleted the lines below that went with "br" type of stream.
           
            // rather than read bytes, let's read lines, since the header part is all guaranteed to be lines.
            // we'll read the entire header block up to the blank line separator.
           
            line = readLine(is);
           
            if (debug)
                    System.out.println("Ret Code: " + line);

            String[] split = line.split(" ");
           
            serverRev = split[0];
            returnCode = split[1];
            returnMsg = split[2];
           
            //now read headers. don't think too hard about this.
            while ((line= readLine(is)).length() >0) {
                if (debug) System.out.println(line);
                split = line.split(": ");
                receiveprops.setProperty(split[0], split[1]);
            }
            if (debug) System.out.println();
           
            // get the content-length, if specified. If not, default to -1, which we expect means "chunked"
            // the whole reason for this code in the first place.
           
            int len = Integer.parseInt(receiveprops.getProperty("Content-Length", "-1")); // returns -1 even if there's no entry at all
           
            if (debug) System.out.println("CL: " + len);
           
            String sb = null;//new String("");
           
            // this is the empty result
            if (returnCode.equals("204")) { // nothing to see here, folks. move along.               
            }
           
            // normal result
            else if (returnCode.equals("200")) {
                // all's well
                if (len>0) { // normal content, one long byte-sequence.
                    int retry = 0;
                    int num_tries = 5;
                   
                    byte[] bts = new byte[len];
                   
                    ByteArrayOutputStream baos = new ByteArrayOutputStream(len); // we know the size
                   
                    // ok, this might be excessive. for small results, it undoubtedly is. long ones, gotta do it.
                    while (retry++                        if (is.available()>0) {
                            int len2 = is.read(bts);
                            baos.write(bts, 0, len2);
                        } else {
                        // interesting. Adding the println wastes enough time that the socket catches up to the incoming data.
                        // otherwise, above loop goes too fast and terminates before all the data is read.
                        System.out.println("#");
                        }
                    }
                    sb = baos.toString("UTF-8");
                }
               
                // now for chunked encoding
                else if (receiveprops.containsKey("Transfer-Encoding")
                        && receiveprops.getProperty("Transfer-Encoding").equalsIgnoreCase("chunked")) {
                   
                    String chunkSizeLine = readLine(is); // this is a hex value
                   
                    ByteArrayOutputStream baos = new ByteArrayOutputStream();
                   
                    // now we are reading the chunk
                    while (true) {
                        if (debug) System.out.println("chunk size line hex: " + chunkSizeLine);
                       
                        // next is chunk size, which is given in hex.
                        int chunkSize = Integer.parseInt(chunkSizeLine, 16);
                        if (chunkSize == 0) break; // final chunk size is always zero. This not zero? not final chunk.

                        if (debug) System.out.println("chunk size decimal: " + chunkSize);
                       
                        // in general, the chunks aren't large. Doesnt mean they can't be...
                        byte[] buf = new byte[chunkSize];
                        int rlen = 0;
                        // read the entire chunk.
                        while (rlen                            rlen += is.read(buf,rlen,chunkSize-rlen);
                        }
                        baos.write(buf);
                       
                        if (debug) System.out.println("bytes read (=chunk size decimal): " + rlen);
                       
                        // on to the next chunk
                        // read a blank
                        readLine(is);
                        chunkSizeLine= readLine(is);
                        if (debug) System.out.println();
                    }
                   
                    String temp = baos.toString("UTF-8");
                    if (debug) System.out.println(temp);
                   
                    sb = temp; // this is it
                }
            }
           
            // ok, I probably better read an error message
            // I think the server actually creates these errors, when your calling params don't match the method signature
            else if (returnCode.equals("400")
                    || returnCode.equals("404")
                    || returnCode.equals("500")
                    ) {
                if (debug) System.out.println(receiveprops);
               
                if (len>0)  { // normal content, one long byte-sequence. even though it's an error.
                    int retry = 0;
                    int num_tries = 5;
                   
                    byte[] bts = new byte[len];
                   
                    ByteArrayOutputStream baos = new ByteArrayOutputStream(len); // we know the size
                   
                    // ok, this might be excessive. for small results, it undoubtedly is. long ones, gotta do it.
                    while (retry++                        while (is.available()>0) {
                            int len2 = is.read(bts);
                            baos.write(bts, 0, len2);
                        }
                    }
                    sb = baos.toString("UTF-8");
                }
            }
            is.close();
            so.close();
           
            if (debug) System.out.println(sb.length());
           
            return sb;
           
        } catch (MalformedURLException err) {
            err.printStackTrace();
        } catch (UnknownHostException err) {
            err.printStackTrace();
        } catch (IOException err) {
            err.printStackTrace();
        }
        return null; // oops.
    }

    // *******************************************
    // *******************************************
    // *******************************************

    public static void main(String[] args) {
        NanoHTTPClient nano = new NanoHTTPClient();

        // C-L normal return.
        System.out.println(nano.callMethodGet("http://httpbin.org/"));
        System.out.println(nano.callMethodGet("http://httpbin.org/ip"));
       
        //nano.debug = true;
        // YAY! CHUNKED!
        System.out.println(nano.callMethodGet("http://www.google.com/"));
    }
   
}

another java hacking story, part 2

So do it myself means I open the socket, I write the output to the server, then I read the input from the server.

Means I have to write the "readLine" method. Again.

Well, anyway. Nothing I haven't done before. Except the business about "getErrorStream()" in HttpUrlConnection. What is that?

Took me a while to hunt down the right thing, which is the deeply buried Sun code for this class. What happens: normal process is for the header block to be read for you, then you call getInputStream() for the remainder. But if the return code is >=400, the header is read, and then:

InputStream instream = socket.getInputStream();
InputStream errstream = null;

// read header now.

// return code >= 400?

errstream = instream;
instream = null;


There you have it. There's only the one stream, from the socket. IN and ERR are flipped. You can't ever have both at the same time (note that this is different from an exec'd subprocess, but that's OS-level behavior for a process).

So the "content" that gets read from the socket is either the normal result, or it's the error message. Some messages are generated by the server (Tomcat), and some are from the servlet code.

OK, so no big deal to read all this. And yea, verily, with a couple initial errors readily fixed, it works as it is supposed to.

And I have a new tool.

And neither you nor I can trust HttpUrlConnection again. But with a new tool, who needs to?

Part 3 is the source for the final item. It isn't finished, I'm not handling the various other return codes, just those necessary for what I was doing at work. I'll fix up those things as I need to.

another java hacking story, part 1

for somewhat dumb contract-funding reasons, I am working on a new activity for a few months.

this activity is to improve a REST API written by others. we'll skip the reasons and flavor of how bad it is...which is bad. It is mostly functional, but the engineering on it is poor. Example: comments in the code are about 1% of what they ought to be, across ~300 files.

How it's organized:

Oracle--Hibernate--Java classes--JAX-RS--Tomcat--web client

Nothing too unusual about that. I have some Hibernate experience, from 8 years ago, but that too was code I inherited, also functional (and better done) so I didn't have to understand it too deep. I have SQL experience, REST experience, Tomcat experience...etc.so I can do this.

It takes me a while to get the stupid thing to run, because it's very fragile to assemble, and in fact no one knows exactly how to do it. That should tell you a fair amount about its origins--a project that was quite expensive and failed fairly badly. I don't know how many people lost their jobs over it (my employer was uninvolved). Not enough, I expect. Customer wants to probably have less egg on face, so we are carrying a small amount forward. But because of the origins, we don't completely know how to build from scratch. Well, we still don't, in an automated fashion, but I could hand-assemble it from scratch.

There's some amount of JUnit testing built in, when you run Maven to build .war files, but that's all broken. I'm not working that part, yet.

I need a client program to test with, so I can exercise the services thoroughly for a baseline, and then when I am making code changes. But of course there isn't one. No big deal, I have my own that I have been using for years.

I wrote my client program 10-12 years ago, far enough back that I don't actually remember when. I've used it a lot of times, generally with NanoHTTPD (another favorite piece of code, from Jarno Elonen; you can find it online, although I think what you can find now is not as good as it used to be--Jarno appeared to lose control over it several years ago, it got moved to sourceforge or github or something like that, and then go downhill; I have I think his final personal version). It is has always worked fine for me. Until this project.

Apparently all I've done with it is simplistic things. And now I have something harder. It errors out a lot as the client for this REST activity.

I have no idea what is going on, but I didn't bother with it for several weeks, needing to get other things going--but once I reached the point where I needed to be measuring success rates, those failures couldn't be tolerated.

So I dug in to find out what was going on. I've been doing sockets for 30 years, I can do this.

Turned out the failures all occurred when the server (Tomcat) was sending service results back with "Transfer-Encoding: Chunked". Chunked output has "Content-Length: -1" so you can't just read N bytes and be done.

My client program had always been using java.net.HttpUrlConnection -- and guess what? That doesn't quite work right with chunked returns. Well, HttpUrlConnection has been around since what looks like Day 1 for Java, so it predates the existence of "chunked", but you'd think maybe it would have been modified since then? Apparently not. OK, that means I have to do the handling/reading of it--fine, I can read sockets ok.

Or can I? I read up on what "chunked" means in terms of data formatting; it's not complicated, looks straightforward and easy to implement.

Well, yeah, sort of. I mean I wrote something to do it, only that fails every time. So I went back to simpler approach, and simply read all the bytes until nothing left--ahah! There's something missing!

Your normal result should be:

HTTP/1.1 200 OK
header
header

hex-value-of-chunk-length
chunk-bytes

hex-value-of-chunk-length
chunk-bytes

0


HttpUrlConnection is going to read all the header stuff for you, and then you start reading, which will be at the first "hex-value" point. But that seems to be not true, it appears that I am starting just after that. I horsed around with streams back and forth to try to figure out what was going on, to no avail. Could not get it to go.

So since this project has "Spring" around, I thought I would try that, too. Well, that did slightly better, it appeared to be reading all the bytes, but a bunch of times the result would be a big long string of spaces, not my content. WTF? This isn't new code either, but it too doesn't deal with chunked results properly.

So first off: why the chunking? I don't know. This seems to be something Tomcat is doing for me, the services don't appear to have any involvement, so it's not like I can turn it off. I am stuck.

And now I'm annoyed.

[Aside: there's a theme running through my professional career about occasions where I had a problem with someone else's code, and concluded that I would have to rewrite it from scratch in order to get something I could control, and assure that it worked correctly. And I'm the person who can do it.]

So now I'm at the point where I am going to have to rewrite it from scratch.

Saturday, March 05, 2016

some java hacking

For some reason I got motivated to investigate this little problem. Finally. After I don't know how many years...

java.awt.BorderLayout doesn't let you put things in the corners.

The corners don't even exist in their own right.

All that exists are a bar across the top, full width, a similar bar at the bottom, and left/right sections in between those bars, and then the center that resizes to fill the gap.

 But what about when you want those corners?
 This is one approach:


It resembles the desired result, but is fairly hackish by using nested BorderLayouts. Ugh.
 Google "borderlayout corner" and you get a bunch of non-answers, all very hackish, all of which purport to do what you want, but have various drawbacks ("I would use nested JPanels with such-n-such" or some other garbage). Stackoverflow is where most of the discussion takes place, and the result is typical of SOF. GridBag (yuk), the above nested BorderLayouts, "put your widgets into a custom Border", etc; all yuk, all missing the target.

The basic answer is that there's no built-in way in awt/swing to do this. At all. Which is kinda sad.

Actually the custom border is almost the right approach.

-------------------

I had wanted this years ago, long enough back that I don't remember when the idea first occurred to me.

Should not take an Advanced Degree(tm) to figure this out. Altho looking at that SOF discussion, it might. Fortunately that's what we have here at Hyde U.

-------------------

The answer is of course that you have to make your own layout manager. That's not terribly hard, I've done it before, with a grid variation.




The NE corner is a JButton. The other corners are JLabels. The sides are JPanels, with one JLabel each. The center is just a JPanel. All with background colors that are hideous so you can see where the boundaries are. The center/east boundary appears to have a one-pixel glitch I haven't figured out.

In the code below, it's ok to only fill one corner. Remember that JButton uses a lot of margin space, so if you wanted (for example, and the reason I started on this finally) a tiny little "X" button @ NE, you're going to have to force the JButton to be the size you want.

All the proper resizing takes place, layout is dynamically computed to match whatever you are putting in the corners.

Of course this leads to a fairly horrible aesthetic around the sides because the corners can force the sides into sizes you don't like, but the whole thing DOES work. If you want something sized more like the first image, then you should go with that alternative. It's in the code at the bottom.

How'd I do this? I grabbed the Java 7 source for BorderLayout, and started banging on it. It's not quite finished, I want to strip out some extraneous garbage (the LTR stuff, and the PAGE_START stuff), hgap/vgap aren't used everywhere as spacers...

Took me about 3 hours. Longer than it should have, I went down a bad direction with cut-n-paste errors, had to start over.

Here it is:
(note that this is at least partially copyright Sun Microsystems, from way back when; you can find the Java 7 source code easily enough, and see the original I started from)

--------

package com.hu.Test;

import java.awt.Component;
import java.awt.Container;
import java.awt.Dimension;
import java.awt.Insets;
import java.awt.LayoutManager2;

// HU: jeez, I should have written this over a decade ago. Well, finally, here it is.

/**
 * A border layout lays out a container, arranging and resizing
 * its components to fit in five regions:
 * north, south, east, west, and center.
 *
 * This modified version ALSO allows you to use the corners to hold other components. HU, Mar, 2016.
 *
 * Each region may contain no more than one component, and
 * is identified by a corresponding constant:
 * NORTH, SOUTH, EAST,
 * WEST, and CENTER.  When adding a
 * component to a container with a border layout, use one of these
 * five constants, for example:
 *
 *    Panel p = new Panel();
 *    p.setLayout(new BorderLayout());
 *    p.add(new Button("Okay"), BorderLayout.SOUTH);
 * 

 * As a convenience, BorderLayout interprets the
 * absence of a string specification the same as the constant
 * CENTER:
 *
 *    Panel p2 = new Panel();
 *    p2.setLayout(new BorderLayout());
 *    p2.add(new TextArea());  // Same as p.add(new TextArea(), BorderLayout.CENTER);
 * 

 *
 * NOTE: Currently (in the Java 2 platform v1.2),
 * BorderLayout does not support vertical
 * orientations.  The isVertical setting on the container's
 * ComponentOrientation is not respected.
 *
 * The components are laid out according to their
 * preferred sizes and the constraints of the container's size.
 * The NORTH and SOUTH components may
 * be stretched horizontally; the EAST and
 * WEST components may be stretched vertically;
 * the CENTER component may stretch both horizontally
 * and vertically to fill any space left over.
 *
 *
 * @author      Arthur van Hoff (original), Hyde University (corners)
 * @see         java.awt.Container#add(String, Component)
 * @see         java.awt.ComponentOrientation
 * @see            java.awt.BorderLayout
 * @since       JDK1.0
 */
public class BorderLayout implements LayoutManager2, java.io.Serializable {
    /**
     * Constructs a border layout with the horizontal gaps
     * between components.
     * The horizontal gap is specified by hgap.
     *
     * @see #getHgap()
     * @see #setHgap(int)
     *
     * @serial
     */
    int hgap;

    /**
     * Constructs a border layout with the vertical gaps
     * between components.
     * The vertical gap is specified by vgap.
     *
     * @see #getVgap()
     * @see #setVgap(int)
     * @serial
     */
    int vgap;

    /**
     * Constant to specify components location to be the
     *      north portion of the border layout.
     * @serial
     * @see #getChild(String, boolean)
     * @see #addLayoutComponent
     * @see #getLayoutAlignmentX
     * @see #getLayoutAlignmentY
     * @see #removeLayoutComponent
     */
    Component north;
    /**
     * Constant to specify components location to be the
     *      west portion of the border layout.
     * @serial
     * @see #getChild(String, boolean)
     * @see #addLayoutComponent
     * @see #getLayoutAlignmentX
     * @see #getLayoutAlignmentY
     * @see #removeLayoutComponent
     */
    Component west;
    /**
     * Constant to specify components location to be the
     *      east portion of the border layout.
     * @serial
     * @see #getChild(String, boolean)
     * @see #addLayoutComponent
     * @see #getLayoutAlignmentX
     * @see #getLayoutAlignmentY
     * @see #removeLayoutComponent
     */
    Component east;
    /**
     * Constant to specify components location to be the
     *      south portion of the border layout.
     * @serial
     * @see #getChild(String, boolean)
     * @see #addLayoutComponent
     * @see #getLayoutAlignmentX
     * @see #getLayoutAlignmentY
     * @see #removeLayoutComponent
     */
    Component south;
    /**
     * Constant to specify components location to be the
     *      center portion of the border layout.
     * @serial
     * @see #getChild(String, boolean)
     * @see #addLayoutComponent
     * @see #getLayoutAlignmentX
     * @see #getLayoutAlignmentY
     * @see #removeLayoutComponent
     */
    Component center;

    Component northeast, southeast, northwest, southwest;

    /**
     * The north layout constraint (top of container).
     */
    public static final String NORTH  = "North";

    /**
     * The south layout constraint (bottom of container).
     */
    public static final String SOUTH  = "South";

    /**
     * The east layout constraint (right side of container).
     */
    public static final String EAST   = "East";

    /**
     * The west layout constraint (left side of container).
     */
    public static final String WEST   = "West";

    /**
     * The center layout constraint (middle of container).
     */
    public static final String CENTER = "Center";

    public static final String NORTHEAST = "Northeast";
    public static final String NORTHWEST = "Northwest";
    public static final String SOUTHEAST = "Southeast";
    public static final String SOUTHWEST = "Southwest";

    /*
     * JDK 1.1 serialVersionUID
     */
    private static final long serialVersionUID = -8658291919501921765L;

    /**
     * Constructs a new border layout with
     * no gaps between components.
     */
    public BorderLayout() {
        this(0, 0);
    }

    /**
     * Constructs a border layout with the specified gaps
     * between components.
     * The horizontal gap is specified by hgap
     * and the vertical gap is specified by vgap.
     * @param   hgap   the horizontal gap.
     * @param   vgap   the vertical gap.
     */
    public BorderLayout(int hgap, int vgap) {
        this.hgap = hgap;
        this.vgap = vgap;
    }

    /**
     * Returns the horizontal gap between components.
     * @since   JDK1.1
     */
    public int getHgap() {
        return hgap;
    }

    /**
     * Sets the horizontal gap between components.
     * @param hgap the horizontal gap between components
     * @since   JDK1.1
     */
    public void setHgap(int hgap) {
        this.hgap = hgap;
    }

    /**
     * Returns the vertical gap between components.
     * @since   JDK1.1
     */
    public int getVgap() {
        return vgap;
    }

    /**
     * Sets the vertical gap between components.
     * @param vgap the vertical gap between components
     * @since   JDK1.1
     */
    public void setVgap(int vgap) {
        this.vgap = vgap;
    }

    /**
     * Adds the specified component to the layout, using the specified
     * constraint object.  For border layouts, the constraint must be
     * one of the following constants:  NORTH,
     * SOUTH, EAST,
     * WEST, or CENTER,
     * NORTHEAST, NORTHWEST,
     * SOUTHEAST, SOUTHWEST
     *
     * Most applications do not call this method directly. This method
     * is called when a component is added to a container using the
     * Container.add method with the same argument types.
     * @param   comp         the component to be added.
     * @param   constraints  an object that specifies how and where
     *                       the component is added to the layout.
     * @see     java.awt.Container#add(java.awt.Component, java.lang.Object)
     * @exception   IllegalArgumentException  if the constraint object is not
     *                 a string, or if it not one of the five specified
     *              constants.
     * @since   JDK1.1
     */
    public void addLayoutComponent(Component comp, Object constraints) {
        synchronized (comp.getTreeLock()) {
            if ((constraints == null) || (constraints instanceof String)) {
                addLayoutComponent((String)constraints, comp);
            } else {
                throw new IllegalArgumentException("cannot add to layout: constraint must be a string (or null)");
            }
        }
    }

    /**
     * @deprecated  replaced by addLayoutComponent(Component, Object).
     */
    @Deprecated
    public void addLayoutComponent(String name, Component comp) {
        synchronized (comp.getTreeLock()) {
            /* Special case:  treat null the same as "Center". */
            if (name == null) {
                name = "Center";
            }

            /* Assign the component to one of the known regions of the layout.
             */
            if (CENTER.equals(name)) {
                center = comp;
            } else if (NORTH.equals(name)) {
                north = comp;
            } else if (SOUTH.equals(name)) {
                south = comp;
            } else if (EAST.equals(name)) {
                east = comp;
            } else if (WEST.equals(name)) {
                west = comp;

            } else if (NORTHEAST.equals(name)) {
                northeast = comp;
            } else if (SOUTHEAST.equals(name)) {
                southeast = comp;
            } else if (NORTHWEST.equals(name)) {
                northwest = comp;
            } else if (SOUTHWEST.equals(name)) {
                southwest = comp;

            } else {
                throw new IllegalArgumentException("cannot add to layout: unknown constraint: " + name);
            }
        }
    }

    /**
     * Removes the specified component from this border layout. This
     * method is called when a container calls its remove or
     * removeAll methods. Most applications do not call this
     * method directly.
     * @param   comp   the component to be removed.
     * @see     java.awt.Container#remove(java.awt.Component)
     * @see     java.awt.Container#removeAll()
     */
    public void removeLayoutComponent(Component comp) {
        synchronized (comp.getTreeLock()) {
            if (comp == center) {
                center = null;
            } else if (comp == north) {
                north = null;
            } else if (comp == south) {
                south = null;
            } else if (comp == east) {
                east = null;
            } else if (comp == west) {
                west = null;

            } else if (comp == northeast) {
                northeast = null;
            } else if (comp == southeast) {
                southeast = null;
            } else if (comp == northwest) {
                northwest = null;
            } else if (comp == southwest) {
                southwest = null;
            }

        }
    }

    /**
     * Gets the component that was added using the given constraint
     *
     * @param   constraints  the desired constraint, one of CENTER,
     *                       NORTH, SOUTH,
     *                       WEST, EAST,
     *                       NORTHEAST, NORTHWEST,
     *                       SOUTHEAST, SOUTHWEST
     * @return  the component at the given location, or null if
     *          the location is empty
     * @exception   IllegalArgumentException  if the constraint object is
     *              not one of the specified constants
     * @see     #addLayoutComponent(java.awt.Component, java.lang.Object)
     * @since 1.5
     */
    public Component getLayoutComponent(Object constraints) {

        if (CENTER.equals(constraints)) {
            return center;

        } else if (NORTH.equals(constraints)) {
            return north;
        } else if (SOUTH.equals(constraints)) {
            return south;
        } else if (WEST.equals(constraints)) {
            return west;
        } else if (EAST.equals(constraints)) {
            return east;

        } else if (NORTHEAST.equals(constraints)) {
            return northeast;
        } else if (SOUTHEAST.equals(constraints)) {
            return southeast;
        } else if (NORTHWEST.equals(constraints)) {
            return northwest;
        } else if (SOUTHWEST.equals(constraints)) {
            return southwest;

        } else {
            throw new IllegalArgumentException("cannot get component: unknown constraint: " + constraints);
        }
    }


    /**
     * Returns the component that corresponds to the given constraint location.
     *
     * @param   constraints     the desired absolute position, one of CENTER,
     *                          NORTH, SOUTH,
     *                          EAST, WEST,
     *                          NORTHEAST, NORTHWEST,
     *                          SOUTHEAST, SOUTHWEST
     * @param   target     the {@code Container} used to obtain
     *                     the constraint location based on the target
     *                     {@code Container}'s component orientation.
     * @return  the component at the given location, or null if
     *          the location is empty
     * @exception   IllegalArgumentException  if the constraint object is
     *              not one of the specified constants
     * @exception   NullPointerException  if the target parameter is null
     * @see     #addLayoutComponent(java.awt.Component, java.lang.Object)
     * @since 1.5
     */
    public Component getLayoutComponent(Container target, Object constraints) {
        //boolean ltr = target.getComponentOrientation().isLeftToRight();
        Component result = null;

        if (NORTH.equals(constraints)) {
            result = north;
        } else if (SOUTH.equals(constraints)) {
            result = south;
        } else if (WEST.equals(constraints)) {
            result = west;
        } else if (EAST.equals(constraints)) {
            result = east;
       
        } else if (NORTHEAST.equals(constraints)) {
            result = northeast;
        } else if (SOUTHEAST.equals(constraints)) {
            result = southeast;
        } else if (NORTHWEST.equals(constraints)) {
            result = northwest;
        } else if (SOUTHWEST.equals(constraints)) {
            result = southwest;

        } else if (CENTER.equals(constraints)) {
            result = center;

        } else {
            throw new IllegalArgumentException("cannot get component: invalid constraint: " + constraints);
        }

        return result;
    }


    /**
     * Gets the constraints for the specified component
     *
     * @param   comp the component to be queried
     * @return  the constraint for the specified component,
     *          or null if component is null or is not present
     *          in this layout
     * @see #addLayoutComponent(java.awt.Component, java.lang.Object)
     * @since 1.5
     */
    public Object getConstraints(Component comp) {
        //fix for 6242148 : API method java.awt.BorderLayout.getConstraints(null) should return null
        if (comp == null){
            return null;
        }
        if (comp == center) {
            return CENTER;
        } else if (comp == north) {
            return NORTH;
        } else if (comp == south) {
            return SOUTH;
        } else if (comp == west) {
            return WEST;
        } else if (comp == east) {
            return EAST;

        } else if (comp == northwest) {
            return NORTHWEST;
        } else if (comp == southwest) {
            return SOUTHWEST;
        } else if (comp == northwest) {
            return NORTHWEST;
        } else if (comp == southwest) {
            return SOUTHWEST;
        }
        return null;
    }

    /**
     * Determines the minimum size of the target container
     * using this layout manager.
     *
     * This method is called when a container calls its
     * getMinimumSize method. Most applications do not call
     * this method directly.
     * @param   target   the container in which to do the layout.
     * @return  the minimum dimensions needed to lay out the subcomponents
     *          of the specified container.
     * @see     java.awt.Container
     * @see     java.awt.BorderLayout#preferredLayoutSize
     * @see     java.awt.Container#getMinimumSize()
     */
    public Dimension minimumLayoutSize(Container target) {
        synchronized (target.getTreeLock()) {
            Dimension dim = new Dimension(0, 0);

            boolean ltr = target.getComponentOrientation().isLeftToRight();
            Component c = null;

            if ((c=getChild(EAST,ltr)) != null) {
                Dimension d = c.getMinimumSize();
                dim.width += d.width + hgap;
                dim.height = Math.max(d.height, dim.height);
            }
            if ((c=getChild(WEST,ltr)) != null) {
                Dimension d = c.getMinimumSize();
                dim.width += d.width + hgap;
                dim.height = Math.max(d.height, dim.height);
            }
            if ((c=getChild(CENTER,ltr)) != null) {
                Dimension d = c.getMinimumSize();
                dim.width += d.width;
                dim.height = Math.max(d.height, dim.height);
            }
            if ((c=getChild(NORTH,ltr)) != null) {
                Dimension d = c.getMinimumSize();
                dim.width = Math.max(d.width, dim.width);
                dim.height += d.height + vgap;
            }
            if ((c=getChild(SOUTH,ltr)) != null) {
                Dimension d = c.getMinimumSize();
                dim.width = Math.max(d.width, dim.width);
                dim.height += d.height + vgap;
            }

            if ((c=getChild(NORTHEAST,ltr)) != null) {
                Dimension d = c.getMinimumSize();
                dim.width += d.width + hgap;
                dim.height = d.height + vgap;
            }

            Insets insets = target.getInsets();
            dim.width += insets.left + insets.right;
            dim.height += insets.top + insets.bottom;

            return dim;
        }
    }

    /**
     * Determines the preferred size of the target
     * container using this layout manager, based on the components
     * in the container.
     *
     * Most applications do not call this method directly. This method
     * is called when a container calls its getPreferredSize
     * method.
     * @param   target   the container in which to do the layout.
     * @return  the preferred dimensions to lay out the subcomponents
     *          of the specified container.
     * @see     java.awt.Container
     * @see     java.awt.BorderLayout#minimumLayoutSize
     * @see     java.awt.Container#getPreferredSize()
     */
    public Dimension preferredLayoutSize(Container target) {
        synchronized (target.getTreeLock()) {
            Dimension dim = new Dimension(0, 0);

            boolean ltr = target.getComponentOrientation().isLeftToRight();
            Component c = null;

            if ((c=getChild(EAST,ltr)) != null) {
                Dimension d = c.getPreferredSize();
                dim.width += d.width + hgap;
                dim.height = Math.max(d.height, dim.height);
            }
            if ((c=getChild(WEST,ltr)) != null) {
                Dimension d = c.getPreferredSize();
                dim.width += d.width + hgap;
                dim.height = Math.max(d.height, dim.height);
            }
            if ((c=getChild(CENTER,ltr)) != null) {
                Dimension d = c.getPreferredSize();
                dim.width += d.width;
                dim.height = Math.max(d.height, dim.height);
            }
            if ((c=getChild(NORTH,ltr)) != null) {
                Dimension d = c.getPreferredSize();
                dim.width = Math.max(d.width, dim.width);
                dim.height += d.height + vgap;
            }
            if ((c=getChild(SOUTH,ltr)) != null) {
                Dimension d = c.getPreferredSize();
                dim.width = Math.max(d.width, dim.width);
                dim.height += d.height + vgap;
            }

            if ((c=getChild(NORTHEAST,ltr)) != null) {
                Dimension d = c.getPreferredSize();
                dim.width += d.width + hgap;
                dim.height = d.height + vgap;
            }

            Insets insets = target.getInsets();
            dim.width += insets.left + insets.right;
            dim.height += insets.top + insets.bottom;

            return dim;
        }
    }

    /**
     * Returns the maximum dimensions for this layout given the components
     * in the specified target container.
     * @param target the component which needs to be laid out
     * @see Container
     * @see #minimumLayoutSize
     * @see #preferredLayoutSize
     */
    public Dimension maximumLayoutSize(Container target) {
        return new Dimension(Integer.MAX_VALUE, Integer.MAX_VALUE);
    }

    /**
     * Returns the alignment along the x axis.  This specifies how
     * the component would like to be aligned relative to other
     * components.  The value should be a number between 0 and 1
     * where 0 represents alignment along the origin, 1 is aligned
     * the furthest away from the origin, 0.5 is centered, etc.
     */
    public float getLayoutAlignmentX(Container parent) {
        return 0.5f;
    }

    /**
     * Returns the alignment along the y axis.  This specifies how
     * the component would like to be aligned relative to other
     * components.  The value should be a number between 0 and 1
     * where 0 represents alignment along the origin, 1 is aligned
     * the furthest away from the origin, 0.5 is centered, etc.
     */
    public float getLayoutAlignmentY(Container parent) {
        return 0.5f;
    }

    /**
     * Invalidates the layout, indicating that if the layout manager
     * has cached information it should be discarded.
     */
    public void invalidateLayout(Container target) {
    }

    /**
     * Lays out the container argument using this border layout.
     *
     * This method actually reshapes the components in the specified
     * container in order to satisfy the constraints of this
     * BorderLayout object. The NORTH
     * and SOUTH components, if any, are placed at
     * the top and bottom of the container, respectively. The
     * WEST and EAST components are
     * then placed on the left and right, respectively. Finally,
     * the CENTER object is placed in any remaining
     * space in the middle.
     *
     * And the corners work as well, with NORTHEAST, NORTHWEST, SOUTHEAST, SOUTHWEST.
     *
     * Most applications do not call this method directly. This method
     * is called when a container calls its doLayout method.
     * @param   target   the container in which to do the layout.
     * @see     java.awt.Container
     * @see     java.awt.Container#doLayout()
     */
    public void layoutContainer(Container target) {
        synchronized (target.getTreeLock()) {
            Insets insets = target.getInsets();
            int top = insets.top;
            int bottom = target.getHeight() - insets.bottom;
            int left = insets.left;
            int right = target.getWidth() - insets.right;

            boolean ltr = target.getComponentOrientation().isLeftToRight();
            Component c = null;

            // order is somewhat important in the original, because there was an implied ordering to what was going on, top->bottom, L->R

            // now there's not.
           
            // we have to manage all 8 edge pieces together.
           
            Dimension zero = new Dimension(0, 0);
           
            Dimension nw = northwest != null ? northwest.getPreferredSize() : zero;
            Dimension no = north != null ? north.getPreferredSize() : zero;
            Dimension ne = northeast != null ? northeast.getPreferredSize() : zero;

            Dimension we = west != null ? west.getPreferredSize() : zero;
            Dimension ea = east != null ? east.getPreferredSize() : zero;

            Dimension sw = southwest != null ? southwest.getPreferredSize() : zero;
            Dimension so = south != null ? south.getPreferredSize() : zero;
            Dimension se = southeast != null ? southeast.getPreferredSize() : zero;
                       
            // boundary sizes.
           
            int lwid = Math.max(Math.max(nw.width, we.width), sw.width);
            int rwid = Math.max(Math.max(ne.width, ea.width), se.width);
            int thei = Math.max(Math.max(nw.height, no.height), ne.height);
            int bhei = Math.max(Math.max(sw.height, so.height), se.height);
           
            // each of these methods resizes the widget: setSize, setBounds

            if ((c=getChild(NORTH,ltr)) != null) {
                c.setBounds(left+lwid, top, right - left - lwid - rwid, thei);
            }
                       
            if ((c=getChild(SOUTH,ltr)) != null) {
                c.setBounds(left+lwid, bottom - bhei, right - left - lwid - rwid, bhei);
            }
           
            if ((c=getChild(EAST,ltr)) != null) {
                c.setBounds(right - rwid, top+thei, rwid, bottom - top - thei - bhei);
            }
           
            if ((c=getChild(WEST,ltr)) != null) {
                c.setBounds(left, top+thei, lwid, bottom - top - thei - bhei);
            }
           
            if ((c=getChild(CENTER,ltr)) != null) {
                c.setBounds(left + lwid, top + thei, right - left - rwid - lwid, bottom - top - thei - bhei);
            }
           
            if ((c=getChild(NORTHEAST,ltr)) != null) {
                c.setBounds(right-rwid, top, rwid, thei);
            }
           
            if ((c=getChild(SOUTHEAST,ltr)) != null) {
                c.setBounds(right-rwid, bottom - bhei, rwid, bhei);
            }
           
            if ((c=getChild(NORTHWEST,ltr)) != null) {
                c.setBounds(left, top, lwid, thei);
            }
            if ((c=getChild(SOUTHWEST,ltr)) != null) {
                c.setBounds(left, bottom-bhei, lwid, bhei);
            }
        }
    }

    /**
     * Get the component that corresponds to the given constraint location
     *
     * @param   key     The desired absolute position,
     *                  either NORTH, SOUTH, EAST, or WEST.
     *                  Or NORTHEAST, NORTHWEST, SOUTHEAST, SOUTHWEST.
     * @param   ltr     Is the component line direction left-to-right?
     */
    private Component getChild(String key, boolean ltr) {
        Component result = null;

        if (key == NORTH) {
            result = north;
        }
        else if (key == SOUTH) {
            result = south;
        }
        else if (key == WEST) {
            result = west;
        }
        else if (key == EAST) {
            result = east;
        }

        else if (key == NORTHWEST) {
            result = northwest;
        }
        else if (key == SOUTHWEST) {
            result = southwest;
        }
        else if (key == NORTHEAST) {
            result = northeast;
        }
        else if (key == SOUTHEAST) {
            result = southeast;
        }

        else if (key == CENTER) {
            result = center;
        }
        if (result != null && !result.isVisible()) {
            result = null;
        }
        return result;
    }

    /**
     * Returns a string representation of the state of this border layout.
     * @return    a string representation of this border layout.
     */
    public String toString() {
        return getClass().getName() + "[hgap=" + hgap + ",vgap=" + vgap + "]";
    }
}

--------------------
Here's the test case:

(There are four different approaches in here:)

package com.hu.Test;

//import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.GridBagConstraints;
import java.awt.GridBagLayout;
import java.awt.Insets;

import javax.swing.GroupLayout;
import javax.swing.GroupLayout.Alignment;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;

public class BorderCorners {

    public static void main(String[] args) {
       
        JPanel mainPanel = new JPanel(new GridBagLayout());

        GridBagConstraints gbc = new GridBagConstraints(0, 0, 2,2, 1.0, 1.0,
                GridBagConstraints.NORTHWEST, GridBagConstraints.NONE, new Insets(
                        0, 0, 0, 0), 0, 0);

        mainPanel.add(new JLabel("Left Upper"), gbc);

        gbc.gridx = 2;
        gbc.gridy = 0;
        gbc.anchor = GridBagConstraints.NORTHEAST;
        mainPanel.add(new JLabel("Right Upper"), gbc);

        gbc.gridx = 0;
        gbc.gridy = 2;
        gbc.anchor = GridBagConstraints.SOUTHWEST;
        mainPanel.add(new JLabel("Left Lower"), gbc);

        gbc.gridx = 2;
        gbc.gridy = 2;
        gbc.anchor = GridBagConstraints.SOUTHEAST;
        mainPanel.add(new JLabel("Right Lower"), gbc);

        JFrame frame = new JFrame("Grid Bag Test");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.getContentPane().add(mainPanel);

        // well, this ends up in the middle all right, but it doesn't size up/down when you resize the frame.
        JPanel centerPanel = new JPanel();
        centerPanel.setBackground(Color.red);
        gbc.gridx = 1;
        gbc.gridy = 1;
        gbc.anchor = GridBagConstraints.CENTER;
        mainPanel.add(centerPanel, gbc);


        frame.setLocationRelativeTo(null);
        frame.pack();
        frame.setVisible(true);

        // ***********************

        //yep, this misses the target too.

        JPanel northPanel = new JPanel(new java.awt.BorderLayout());     
        northPanel.add(new JLabel("North East"), java.awt.BorderLayout.EAST);
        northPanel.add(new JLabel("North West"), java.awt.BorderLayout.WEST);

        JPanel southPanel = new JPanel(new java.awt.BorderLayout());
        southPanel.add(new JLabel("South East"), java.awt.BorderLayout.EAST);
        southPanel.add(new JLabel("South West"), java.awt.BorderLayout.WEST);


        mainPanel = new JPanel(new java.awt.BorderLayout());
        mainPanel.add(northPanel, java.awt.BorderLayout.NORTH);
        mainPanel.add(southPanel, java.awt.BorderLayout.SOUTH);

        centerPanel = new JPanel();
        centerPanel.setBackground(Color.red);
        mainPanel.add(centerPanel, java.awt.BorderLayout.CENTER);

        JPanel left = new JPanel();
        left.setBackground(Color.YELLOW);
        mainPanel.add(left, java.awt.BorderLayout.WEST);

        JPanel right = new JPanel();
        right.setBackground(Color.GREEN);
        mainPanel.add(right, java.awt.BorderLayout.EAST);
       
        frame = new JFrame("AWT BorderLayout Test");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.getContentPane().add(mainPanel);
        frame.setLocationByPlatform(true);
       
        frame.setSize(300, 300);
        frame.setVisible(true);

        // ***********************

        System.out.println("trying now with a copy of border layout");

        mainPanel = new JPanel(new BorderLayout());

        JButton xx = new JButton("X");
        System.out.println(xx.getInsets());
        Insets xxi = xx.getInsets();
        xxi.set(2, 2, 2, 2);
        xx.setMargin(xxi);
        //JLabel xx = new JLabel("X");
        xx.setOpaque(true);
        xx.setBackground(Color.MAGENTA);
        xx.setForeground(Color.white);

        mainPanel.add(xx, BorderLayout.NORTHEAST);
        mainPanel.add(new JLabel("North W"), BorderLayout.NORTHWEST);
        mainPanel.add(new JLabel("South East"), BorderLayout.SOUTHEAST);
        mainPanel.add(new JLabel("South W"), BorderLayout.SOUTHWEST);

        northPanel = new JPanel();
        northPanel.setBackground(Color.PINK);
        northPanel.add(new JLabel("N"));
        mainPanel.add(northPanel, BorderLayout.NORTH);

        southPanel = new JPanel();
        southPanel.setBackground(Color.CYAN);
        southPanel.add(new JLabel("S"));
        mainPanel.add(southPanel, BorderLayout.SOUTH);

        centerPanel = new JPanel();
        centerPanel.setBackground(Color.red);
        //mainPanel.add(centerPanel, BorderLayout.CENTER);
        mainPanel.add(centerPanel);
       
        left = new JPanel();
        left.setBackground(Color.YELLOW);
        left.add(new JLabel("W"));
        mainPanel.add(left, BorderLayout.WEST);

        right = new JPanel();
        right.setBackground(Color.GREEN);
        right.add(new JLabel("E"));
        mainPanel.add(right, BorderLayout.EAST);

        frame = new JFrame("NEW BorderLayout Test");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.getContentPane().add(mainPanel);
        frame.pack();
        frame.setLocationByPlatform(true);
        frame.setVisible(true);

        frame.setSize(300, 300);
        frame.setVisible(true);
       
        // ***********************

        // this isn't right either. but it's slightly better.
        frame = new JFrame("Group L");


        JLabel labelNW = new JLabel("NW");
        JButton labelNE = new JButton("NE");
        JLabel labelSW = new JLabel("SW");
        JLabel labelSE = new JLabel("SE");


        GroupLayout layout = new GroupLayout(frame.getContentPane());
        layout.setHorizontalGroup(layout.createSequentialGroup()
                .addGroup(layout.createParallelGroup(Alignment.LEADING)
                        .addComponent(labelNW)
                        .addComponent(labelSW))
                        .addGap(20,50,Short.MAX_VALUE)
                        .addGroup(layout.createParallelGroup(Alignment.TRAILING)
                                .addComponent(labelNE)
                                .addComponent(labelSE))
                );
        layout.setVerticalGroup(layout.createSequentialGroup()
                .addGroup(layout.createParallelGroup(Alignment.LEADING)
                        .addComponent(labelNW)
                        .addComponent(labelNE))
                        .addGap(20,50,Short.MAX_VALUE)
                        .addGroup(layout.createParallelGroup(Alignment.TRAILING)
                                .addComponent(labelSW)
                                .addComponent(labelSE))
                );
        frame.getContentPane().setLayout(layout);
        frame.getContentPane().setBackground(Color.LIGHT_GRAY);
        frame.setSize(200, 200);
        frame.setVisible(true);

    }
}