Making DDS Beg for Mercy

I am sitting and staring at my terminal emulator. I’ve successfully migrated our folks to Initiative Media, who happens to occupy an entire floor of 5700 Wilshire. 5700 Wilshire is a pretty famous Los Angeles office building. It’s not really anything special, but is very modern looking. It has stacked balconies for smokers or execs with personal ones. It’s a mini campus with four towers where the east and west towers are connected on the lower floors and detached in the upper floors. Given that, it’s been in any number of TV and films. It was somebody’s workplace on Melrose Place. Spelling was in the building and used it for his show. If you’ve seen Demolition Man, 5700 Wilshire is where:

• Wesley Snipes’ character beats up security guards in front of a futuristic ATM.
• Sylvester Stallone and Sandra Bullock go down into the otherworld. Sandra says “Let’s go blow these guys.” Sly responds with “Blow these guys away!” She replies “Whatever” and drops down the manhole.

I hear a slight knock and start to get up. The folks at Initiative wanted me to have an office but there was none left. They gave up a breakout room and put me in there. It was very nice, but it also was next to the door leading to the bathroom. People forgot their keycards and I was closest to opening the door. I would do this about seven or eight times a day. About hourly so, I’m still thinking as I walk, open the door, and let the slightly embarrassed person on the other side back in.

The screen that started it. Buy lines keyed by hand. Cursor blinking in the empty space below.

I sit back down and stare at the screen. The green text does not care that I just played doorman. It is waiting. I have a buy order to key — four spots on KTTV, each a different daypart, each with its own rate and demo and length and flight week — and the only way to get them into the system is to type every field by hand, in the right order, on the right screen, and hit Enter, and wait, and do it again. There is no batch import. There is no API. There is no clipboard. There is a keyboard and there is me and there is this screen, and the screen wins every time.

Since I started buying media in 1990, I’ve always been able to extract data. If a system didn’t have a report builder, it would have reports to scrape. I wanted to be able to change data and get dynamic research from the system without a report in the way. Most systems had “file load” capabilities, but they were hobbled as soon as any stateful information got into the system — no loading a “new buy” when the other one is associated to a financial payment.

DDS was currently scrambling. Mainframe interfaces like this, in the late ‘90s, were old. Now that everyone either had a Mac or was running Windows 95+, you needed drop-downs and menus and all sorts of GUI elements to be not just bleeding edge but current. DDS solves this problem by making what is called a fat client, but secretly, the fat client manipulates a typical mainframe session. Nothing new is built — just a fancy UI over the same stuff I’ve been doing since 1993 or so.

This “fat client” exposes something called ActiveX DLLs. There are libraries open to anyone really to control this fat client somehow, but I know the one I want. It’s a single DLL that controls the actual mainframe connection. It exposes exactly one function.

Here is the thing about a 3270 terminal that the fat client is trying to hide from you. The screen is not a window. It is a grid. Twenty-four rows, eighty columns, nineteen hundred and twenty character positions — every one of them numbered, every one of them fixed. Row 6, column 75 is always row 6, column 75. It does not reflow. It does not scroll. It does not care what resolution your monitor is. The mainframe put a character there, or it didn’t, and if it did, that character is sitting in the same position it has occupied since the day the screen was designed, probably sometime in the 1970s.

The fat client dresses this up. It draws boxes around the fields. It adds color. It makes it look like software. But underneath the costume, the screen is still what it always was — a flat buffer, nineteen hundred and twenty bytes, readable in one call.

No parsing. No DOM. No scraping. You know where every field is because it has always been there. You read position 481 because position 481 is the first character of row 7, and row 7 is where the buy command goes, and it has always been where the buy command goes.

This is what the fat client was trying to bury. Not because it was embarrassed by the mainframe — because its job was to make the mainframe feel like Windows. But the mainframe was never trying to be Windows. It was trying to be a fixed-width record with a cursor. And a fixed-width record with a cursor is something a program can read perfectly, every time, without asking permission.

I’ve known of HLLAPI for some time now. IBM’s website helpfully tells me it stands for High Level Language API. I hear that I can use it to both see what the screen shows but also control it somehow. This is the path, but how do I use this one function: hllapi(func_code, instr_buffer, outstr_buffer)?

What is a func_code? Do I really need a “buffer” or just a string variable passed? It’s an ActiveX control so I assume anything that can use ActiveX is good?

For years I search on HLLAPI, and each time, I get close, but there’s always another link to the hidden specification. These links insist that I must be an IBM customer or similar to get access. I am none of these things, but I try various ways over the years. Calling IBM. Ordering online. Nobody responds. So, on a monthly basis, I just try Google.

I search, and something new appears for the first time. A company has recently put their emulator software up for free and — look here. The full HLLAPI spec is here. Holy crap.

I read it and scan through it quickly. “func_code” is like a dispatcher. Pick the right “code” and you get different functions responding.

Let’s try the first one that looks interesting. “CopyPresentationSpace” — I give it two text vars as arguments. It runs immediately. My instr_buffer is filled with data. What is it? It’s the entire contents of the emulator I’m staring at as one long string. Totally awesome, but doesn’t do me much better yet. The real test is next.

One of the func_codes is “send_keys.” I move my cursor on the emulator to the command palette. I issue this fateful call to send_keys: “are you my b-word now.” I watch in wonder as DDS now displays on the screen “are you my b-word now.” I issue one more test — a send_keys with an enter. I watch as DDS complains about my input, and I EXPLODE IN A HAPPY DANCE in my little office.

Nobody opens the bathroom door. Nobody witnesses the dance. This is fine. Some things are between you and the machine.

I build my first HLLAPI-based tool that morning. It’s a simple Excel workbook with VBA in the back. VBA is referencing the ActiveX HLLAPI function so it can call out to my emulator. I build a mini DSL for using SEND_KEYs. Each key would be separated by tildes — never used by DDS itself — and special keys like TAB or HOME had @TAB and @HOME so it was clear. Inputting the header data would look like @HOME~@TAB~T~@TAB~SUNK~@TAB~COLA~@TAB and so on. The first column of the spreadsheet was to be filled by this DSL data. The second column was an AWAIT string. If the AWAIT string didn’t come back from the implicit ENTER at the end of each cell’s contents, the script would stop and ask for help.

Column A: the tilde-separated DSL that HLLAPI executes. Column B: what the screen must say back. If B doesn’t match — the script stops and asks for help.

The first time it runs clean on a real buy, I watch the screen the way you watch a car you just fixed pull out of the driveway. Four commands appear in the palette. Enter. The asterisks land. The numbered lines come up. The header fills in. It takes about four seconds. I had been doing this by hand since 1990. A buy that used to take the better part of an afternoon now takes four seconds and my full attention is no longer required.

The AWAIT string is the part that makes it real. Anyone can write a script that fires keystrokes into a terminal. The script that knows when to stop and ask for help — that is a different thing. That is the script you can actually trust with a live buy on a real account. I add AWAIT strings for every screen transition. If DDS comes back with anything other than what I expect, the whole thing parks and blinks at me. It does not guess. It does not proceed. It waits.

HLLAPI driving DDS. All four commands staged. One Enter. The entire screen updates at once — commands asterisked, buy lines numbered, header filled.

The basic tool is far too complex to distribute, but the pattern is established. I can do things to DDS that no other buyer on the planet can execute — as far as I know. This becomes critical to my value to the company as I am able to make mass transformations that would take the entire group weeks and would be error-prone. I take hours at most and the results are perfect.

At one point, I’m required to make a massive change. All of a given big box retailer’s entire yearly buy needed to go in. There was a template buy and I need to both duplicate this, but also adjust the schedules each week depending on which days the retailer wanted active. This was a huge effort to do by hand and check. My script made the external transformation perfect and ready to get into DDS immediately.

I look at the amount of records that need to go up, multiply by the number of records I need to transmit, and see that this will take more than a day to complete if my machine is the only one working on the problem.

So, as people leave for the day, I ask if I can use their computer. They say I can, and I log in as myself — but LUs are handed out by IP address. I get another session for me. Divide up the work and start this session in parallel. Get more people leaving down the hall. More sessions for me to pound.

At about five parallel sessions, I come back to mine and see that I’ve been kicked off the mainframe. No worries. I log back in and fire the script again. I’m at a neighbor’s machine and see the same thing. It’s clear now. DDS is trying to throttle me. Well, that’s a fun game. I get VNC running on each machine and go back to my station. I can see as each machine is getting pulled off and simply restart them as needed to keep my bandwidth up.

The job eventually completes as I fight with DDS for a few hours.

18,420 pages printed. DDS paper supply: zero. PAPER LOW light is new.

On Monday, my boss calls. DDS is complaining. Evidently they print out all inputs. My script has caused the printer to run out of paper and people are frantic to get it running again in order for the buffer to clear. My boss laughs and says “good job.”

This is the thing about systems designed for humans. Somewhere in the design, there is a budget. A budget for how fast input arrives, how many sessions run at once, how much paper a day’s work requires. The budget is generous — humans are slow, humans need breaks, humans go home at five. The budget is not generous enough for a loop that does not eat, does not get tired, and does not care what time it is.

The printer is the tell. Nobody at DDS ever thought to ask: what if the inputs never stop? The question doesn’t come up in the design meeting because the people in the design meeting are people. They budget for people. I was no longer a person in this context. I was a loop that happened to have a login.

I learned two things from HLLAPI that I have not unlearned since. First: the interface is not the system. The fat client, the green screen, the tab stops, the hand-keyed B commands — none of that was DDS. DDS was the data behind it. Once I could touch the data directly, the interface became irrelevant. Second: when you find the minimum surface area — the one function, the two calls, the nineteen hundred and twenty bytes — you do not need anything else. You do not need menus. You do not need a manual. You need func=3 and func=8 and the knowledge of where everything sits.

That is still how I think about every system I touch. Find the glass. Read it directly. The rest is decoration.

← All case studies · ← Start here: Philosophy to Engineering · Next: The Bill Berger Special