another java hacking story, part 1

for somewhat dumb contract-funding reasons, I am working on a new activity for a few months.

this activity is to improve a REST API written by others. we'll skip the reasons and flavor of how bad it is...which is bad. It is mostly functional, but the engineering on it is poor. Example: comments in the code are about 1% of what they ought to be, across ~300 files.

How it's organized:

Oracle--Hibernate--Java classes--JAX-RS--Tomcat--web client

Nothing too unusual about that. I have some Hibernate experience, from 8 years ago, but that too was code I inherited, also functional (and better done) so I didn't have to understand it too deep. I have SQL experience, REST experience, Tomcat experience...etc.so I can do this.

It takes me a while to get the stupid thing to run, because it's very fragile to assemble, and in fact no one knows exactly how to do it. That should tell you a fair amount about its origins--a project that was quite expensive and failed fairly badly. I don't know how many people lost their jobs over it (my employer was uninvolved). Not enough, I expect. Customer wants to probably have less egg on face, so we are carrying a small amount forward. But because of the origins, we don't completely know how to build from scratch. Well, we still don't, in an automated fashion, but I could hand-assemble it from scratch.

There's some amount of JUnit testing built in, when you run Maven to build .war files, but that's all broken. I'm not working that part, yet.

I need a client program to test with, so I can exercise the services thoroughly for a baseline, and then when I am making code changes. But of course there isn't one. No big deal, I have my own that I have been using for years.

I wrote my client program 10-12 years ago, far enough back that I don't actually remember when. I've used it a lot of times, generally with NanoHTTPD (another favorite piece of code, from Jarno Elonen; you can find it online, although I think what you can find now is not as good as it used to be--Jarno appeared to lose control over it several years ago, it got moved to sourceforge or github or something like that, and then go downhill; I have I think his final personal version). It is has always worked fine for me. Until this project.

Apparently all I've done with it is simplistic things. And now I have something harder. It errors out a lot as the client for this REST activity.

I have no idea what is going on, but I didn't bother with it for several weeks, needing to get other things going--but once I reached the point where I needed to be measuring success rates, those failures couldn't be tolerated.

So I dug in to find out what was going on. I've been doing sockets for 30 years, I can do this.

Turned out the failures all occurred when the server (Tomcat) was sending service results back with "Transfer-Encoding: Chunked". Chunked output has "Content-Length: -1" so you can't just read N bytes and be done.

My client program had always been using java.net.HttpUrlConnection -- and guess what? That doesn't quite work right with chunked returns. Well, HttpUrlConnection has been around since what looks like Day 1 for Java, so it predates the existence of "chunked", but you'd think maybe it would have been modified since then? Apparently not. OK, that means I have to do the handling/reading of it--fine, I can read sockets ok.

Or can I? I read up on what "chunked" means in terms of data formatting; it's not complicated, looks straightforward and easy to implement.

Well, yeah, sort of. I mean I wrote something to do it, only that fails every time. So I went back to simpler approach, and simply read all the bytes until nothing left--ahah! There's something missing!

Your normal result should be:

HTTP/1.1 200 OK
header
header

hex-value-of-chunk-length
chunk-bytes

hex-value-of-chunk-length
chunk-bytes

0


HttpUrlConnection is going to read all the header stuff for you, and then you start reading, which will be at the first "hex-value" point. But that seems to be not true, it appears that I am starting just after that. I horsed around with streams back and forth to try to figure out what was going on, to no avail. Could not get it to go.

So since this project has "Spring" around, I thought I would try that, too. Well, that did slightly better, it appeared to be reading all the bytes, but a bunch of times the result would be a big long string of spaces, not my content. WTF? This isn't new code either, but it too doesn't deal with chunked results properly.

So first off: why the chunking? I don't know. This seems to be something Tomcat is doing for me, the services don't appear to have any involvement, so it's not like I can turn it off. I am stuck.

And now I'm annoyed.

[Aside: there's a theme running through my professional career about occasions where I had a problem with someone else's code, and concluded that I would have to rewrite it from scratch in order to get something I could control, and assure that it worked correctly. And I'm the person who can do it.]

So now I'm at the point where I am going to have to rewrite it from scratch.