Well, from the looks of things, this is becoming a monthly update instead of the weekly one I had planned, but c’est la vie. This path month hasn’t had any massive breakthroughs, but has been one of steady progress.

Vehicle Assembly
Here’s XA-0.1 as of this morning:

I’ll go into more detail later, but as you can see, we’ve got all the plumbing in as well as the wiring. We’ve also got the main Flight Management System computer in and wired up. At this point, we’re pretty much waiting for the engine computers work and the engine qualification work before we can do a final fitup, leakcheck, and start hold-down firings and then flight tests.

Igniter Work
As of last month’s report, we had just assembled our batch of eight igniters (two per engine) and were ready to start the qualification process. We had planned to do a series of 20 firings each, since we felt that the design was similar enough to each of our previous designs that the purpose of the qualification run was just to catch assembly errors. In spite of our new assembly and testing process, we ran into a lot of issues at first. One instance of a part number that I had incorrectly documented in the CAD models (but had correctly documented elsewhere). One instance of an NPT fitting that had been overtighted by one of our interns (causing a snubber to poke out into the combustion flow) who hadn’t done much work with NPT fittings before and wasn’t sure how tight the stuff was supposed to get. One instance of a design change that I had made to tighten up the packaging which I thought wouldn’t affect igniter performance at all but which ended up having issues. We had a case where the CAD model for the sparkplug that I had been using as part of the igniter assembly model to check the fitup of parts on the engine headend ended up being incorrect. We had previously used the spark plug that matched the model, but during the development process we had switched plugs. I had assumed they were the exact same size, but it turns out that one was almost .75in longer than the other.

After all that we were having better luck, but we would still occasionally have a properly assembled igniter that wasn’t making it all the way through the qualification run. This wasn’t acceptable, so we decided to take a day or two to unfreeze the igniter design and reopen the thing for experimentation. We tried several different things, including going back to the older, smaller spark plugs, changing valve sequencing and timing, changing how the igniter pressure tap was attached, using helicoils to make the IPA solenoid manifold mount stronger, and a few other things. Finally we hit on a change that I had been wanting to try for months that ended up not only making the whole igniter a whole bunch more reliable but also gave it a much bigger and much more stable flame. After that change we did a qualification series of 100 firings to verify the design and the igniter. After 100 consecutive ignitions without no-lights, hard starts, or any problems (and all within about a 25 minute time frame), we felt a lot better about the design and refroze it. We then went through and reupdated the igniter assembly, inspection, and qualification checklists (including upping the number of qualification firings to 100 in a row–especially now that with the automated code and the igniter cooling rig, we can do that in about half an hour) and started requalifying the igniters.

A couple of lessons learned from this whole process:

• Design documentation, even if simple is very important. And if you go out of your way to make it, make sure it is located in a place that you will actually use it.
• Procedures are important. While we still had problems, we were able to sort them out and systematically catch most of them a lot quicker than if we had been just “winging it”.
• If you can possibly avoid it, never make a design change without qualifying it first before doing a production run. Sometimes you just don’t have the luxury, but whenever you do, any design change should be tested before commiting a lot of time and money.
• “Batch and Queue” sucks. Just like any other sort of factory, “lean manufacturing” insights apply in the alt.space world too. Assembling a whole batch of igniters before testing the first one is a great way to make the same mistake a whole bunch of times. Assemble one, test it, qualify it, install it, then go to the next. Or at least have one person assembling while the other is qualifying.
• Operations and Maintenance Logs are a good idea. Randall Clague suggested the idea to me. By taking notes on how firings go, recording the number of cumulative firings and cumulative firing duration on a given igniter, recording problems, dissassembly/inspection results, modifications, repairs, maintenance, and all that, you can get a much better feel for what’s going on with the system. It also starts building an actual written record of which areas are problem areas that need work. There’s an old cliche about the sharpest memory being duller than the dullest pencil, and there’s some truth to it. By having dual signoffs on qualification, dissassembly/reassembly, maintenance, and modifications, you increase the odds of catching stupid problems.
• While freezing a design at some point is important in order to get anything actually done, reopenning it when you get the time to allows you to address issues that have come up since the design freeze, and can be very useful.
• The ability to cheaply and quickly iterate and test out different approaches greatly speeds up the development process
• Whenever using small orifices, put a filter in upstream if you can.

Engine Development
This is one area where we’ve ran into several issues in the past month. As I mentioned in the previous post, after our very first engine iteration (which we started testing almost a year ago now), we had had a fairly smooth engine development experience. We ran into issues along the way with the DAQ, the igniters, and the trailer computers, but the engines themselves had always been quite reliable. The regenerative cooling in particular went far quicker than any of us had expected. We pretty much got a working regen design the very first time we tried it. We then went with a lighter weight design of the Chamber-Saddle-Jacket (CSJ hereafter) variety, similar to the designs flown by XCOR and tested by the Swiss Propulsion Lab. That also worked right from the start. We got the parts in for that right after Space Access, and within a week or two we had the thing firing at full throttle for a 30 second burn.

So we were rather surprised with our recent rash of engine problems. A few days before our hard start that we had previously mentioned, we setup to start throttle mapping our very first production engine. After two perfectly nominal firings, our third firing resulted in a burnthrough downstream of the nozzle throat. Seeing the green flame o’ doom really makes you sick, especially when you’re the one who designed the parts, and you know exactly how expensive they are. We were baffled, since the previous regen designs had shown plenty of cooling margin over a very wide range of operating conditions and mixture ratios, and this design had burned a hole near the throat after only 5 seconds into a 10 second firing. Upon dissassembling the engine, we found our culprit–blockage in one of the cooling channels due to debris from an assembly kludge we used to try compensating for what we thought was a machining tolerance error. We fixed the assembly procedure, and went back through and removed the kludge on all our remaining engines, and then reassembled them according to the new procedure.

The second engine we assembled was the prototype CSJ engine that we had received right after Space Access (and which we knew had succesfully reached steady state previously). Anyhow, as you already know, the last firing of that engine ended up in our worst failure to date–a fairly roudy hard start that caused the bolted end closure to separate, turning our engine into a mortar that succesfully launched our chamber 600 feet. We wanted to get into flight testing engines as quickly as possible, but that was not the way we intended to do it. Dave went over a lot of the details in his post, and I won’t rehash them. The single most frustrating thing is that due to not being able to recover the data traces from that run, we don’t have enough information to conclusively pinpoint the root cause of the failure. We have some likely failure modes but not enough evidence to prove any of them conclusively. As it stands, we have evidence that the igniter was firing, and that IPA was flowing, but that the engine didn’t light, which pretty much leaves the only logical explanation being that the LOX wasn’t flowing when it was supposed to and only started flowing after the chamber was being sprayed full of IPA. By the time we had the trailer back the next day, and had taken apart and inspected all the possible pieces that could have temporarily clogged, there was no evidence anywhere of a problem. We even did an LN2 flow test of the system. No problems. Our most likely candidate is still a temporarily jammed LOX fire valve, but even that isn’t a particularly satisfying explanation because we’re in the desert, and we’ve tested in rain, sleet, and snow before, and never had a problem like this.

The good news is that if the failure really was that the LOX didn’t come on when it should have, that failure mode is pretty much impossible with the way the engines are controlled on the flight vehicle. The PLC in the test trailer commands the LOX valve to open, but doesn’t actually have any way of verifying that the LOX injector has succesfully opened before commanding the fuel to open. The engine computer on the other hand, won’t command the IPA valve open until the igniter is firing and the LOX injector pressure has risen to its target level. Whatever the cause of the lack of LOX flow, if the injector hadn’t reached it’s target pressure, the IPA valve wouldn’t have opened. In order for this system to fail, the LOX sensor would have to fail with the exact target voltage, which is extraordinarily unlikely, if not outright impossible. It would be nice if we actually knew what the root cause of the failure was, but with the way the computer is designed, whatever caused the LOX not to flow would just cause the engine not to light.

However, as a result of the failure, we still took several measures to reduce the odds of future problems. We added lights to the trailer so that even testing at night we could see what was going on. Had we seen that IPA was flowing out of the engine without it lighting, we would’ve aborted, and we probably could have caught the root cause of the failure. We also went back through and removed all the old deprecated hardware from the trailer, retested all of the DAQ wiring, did a bunch of maintenance and repair, added a fire suppression valve to the blast frame (that fills the blast frame area with nitrogen gas to smother any fires inside the frame), and made several changes to our testing procedures.

Within a few days, we were back out testing, this time at our newly bulldozed test site out past the airfield, near Rotary Rocket’s old testing pit. Our goal for the test was just to verify that everything was working and to get enough succesful firings that we felt confident that the engine was back up-to-snuff and ready for throttle testing. We ended up having Denise from the X-Prize Foundation come out for a visit during the testing, escorted by our friends Doug Jones and Randall Clague from XCOR. We had several good firings while our visitors were there and a few more after they left. Then right before the shutdown sequence for our last firing of the day we had another burnthrough.

When we did the post mortem on that engine, we figured out what had happened. It turns out that the “machining tolerance issue” that caused us to make the assembly kludge that clogged the cooling channels and killed our first ship-set engine wasn’t a machining tolerance issue at all–it was a modeling/tolerancing issue on my part. As luck would have it our succesful engine had the exact same design in the critical area, but the machining error actually compensated for enough of my mistake the the engine worked, when it really shouldn’t have as designed.

As it is, I’ve fixed the models and drawings, but due to time constraints, we decided to fall back on the design we had gotten working before Space Access that didn’t have a saddle. It’s a bit heavier but a lot more robust, cheaper, and a lot more tolerant to machining errors. Ironically, had our luck been different and our first CSJ engine failed like this, we likely would have gone back to that route sooner and would probably have four qualified engines ready to go right now. As it is, the replacement parts (and a spare) are out being machined at the moment and should be in our hands soon.

When we get the time to revisit our CSJ design, we’ll probably build one prototype of the fixed design, and then test the snot out of the thing to verify that we’ve beaten that problem into the ground before trying to make another batch of engines. The good news is that we now understand some of the key design issues with our engine a lot better than we previously did, which should help save time and money going forward.

A few observations from our recent engine testing experience:

• Cameras are great for helping figure out what went wrong in an accident, so long as they actually have sufficient lighting.
• If you are using someone controlling a manual SCRAM switch, they need to have a good enough view of what is going on to be able to use it correctly.
• Always build spares. The best way to guarantee that you’ll need one is to not have one.
• It’s possible to gain false intuition from what seem to be similar systems.
• If something appears to fit different from the way you intended, it may be worth investigating it up front to verify that the parts are to spec and that your drawings accurately reflect your design intent.
• Having in-house machining capabilities is extremely important. Had we had a machinist and a good CNC mill and lathe, we probably could’ve been back out testing my hypothesis within 1-2 days. Even the best machine shop in the world (and our suppliers at Protocision, Caztek, and I&S machining have been excellent to work with) can’t give you the turn-time that in-house machining can. Having the ability to test easily isn’t enough if it takes you 2-3 weeks to get a new engine in. This is one of those things that we’re going to fix real soon.

Wiring, Electronics, Flight Controls, and Code Development
The thing we’ve been spending most of our time on while waiting for the replacement engines to show up has been getting the electrical and control systems for the vehicle ready to go. As it is, it is probably a good thing that we have the time now due to all the issues we’ve been slowly slogging through on the controls side of things. The assembler for our engine computer boards ended up taking much longer than expected and made several parts placement errors on the boards. Our next door neighbor here in Mojave is an electronics wiz and ended up helping us fix the mistakes for a very reasonable price, but fighting issues with incorrect board design or assembly has cost us several days of debugging and several dead pressure transducers. In spite of all of that, we’ve made a lot of progress on this front. Since electronics and computer programming are mostly black magic to me (I just design rocket engines), I’ll just cover some of the highlights of what we’ve gotten done:

• We’ve completely wired up the vehicle, including power and ground wiring to each of the computers and actuators, ruggedized ethernet connections to each of the engine computers, and SCRAM wiring to each of the computers.
• We’ve got the Flight Management System working. This is a computer that takes commands from the pilot via wireless and uses it to control valves and start the engines. We’ve verified functionality of this system and used it to control all the valves and check all the sensors other than those on the engine modules, which are controlled by the engine computers.
• We’ve got the vehicle network up and running to where the FMS can properly contact each of the engine computers and send them information.
• We’ve got the hinge actuator code working and wrapped into the overall engine control code.
• We’ve gotten the engine computer to fire the igniters correctly.
• We found a way to get the throttle valves moving, implemented our planned throttling scheme, and verified using cold-flow tests that they quickly seek in on the target pressures and have very high accuracy. In fact, we found that they make really good gas pressure regulators when we ran out of water in the pressure tank we were using to test the valve code.
• In spite of not actually getting to the specific throttle mapping tests we had intended to do, because of the scheme we’re using for throttling, I was able to go back through several of our older tests, pull out the data, and construct the lookup tables we needed. If we want to get performance back up, we’ll probably need to individually tune these tables after the fact, but this initial set of look-up tables should be good enough for our first few flights.
• We’ve wrapped all of the gimbal, igniter, and valve codes back into the overall engine control code, and using our cold-flow setup, the igniter cart, and an ignition detection “spoof switch” have gotten it to step correctly through the startup sequence. We found that with the higher feed pressures, we needed to tweak the timing a bit to make sure that the throttle valves didn’t keep hunting back and forth around the target, but we should have that wrapped tonight.
• Tim Bendell from Frontier came last week, and worked with getting the ACS computer integrated with the rest of the vehicle network. We’ve got the mounting worked out, but they had something they still needed to finish up before we could install the hardware onto the vehicle for good.
• We’ve fixed the grounding scheme on our vehicle to make the system a lot more reliable and to greatly reduce the amount of electrical noise from our high-power systems that shows up on our sensors.
• We found the glitch that had been frying our pressure transducers on a disturbingly frequent basis, and fixed it. The code has a “limp home mode” that can handle multiple sensors dying and still keep the engine able to function on some level. At some point you could kill enough sensors that the engine would have to be shutoff, but by that point the pilot should already be on the ground.

This, by the way, is the business end of the throttle testing rig that we put together to verify the engine code functionality. The setup is probably Mojave’s most expensive automated water fountains, but it has made debugging the engine control code and algorithms so much easier than trying to do this on the vehicle or in the trailer:

All in all, in spite of lots of lost hair (I was worried for a while that Pierce and Ian were going to go prematurely bald if this kept up), computer board rework, and code debugging, we’re finally almost through this phase. Our next step once we finish getting the timing tweaked on the valve throttling will be to finish wiring up one of our test engines in the trailer, and doing an end-to-end coldflow test using LN2 instead of LOX. After that, we’re going to try lighting one of our older test engines using the engine control computers and the actual flight valves and flight code. So far all of our tests have been using PLC controls to actuate the actual fire valves. The throttle valves were upstream and were preset before each firing. Starting with that test, we will be testing the engines in a fashion that is as identical as possible to the way they will be fired on the vehicle, other than being horizontal instead of vertical. Once we have a large number of consecutive succesful engine start-ups and shutdowns, we’ll then verify that the engines go into runmode correctly, then try and start controlling the throttling in real-time and verifying the response time of the throttling system. Along the way we’ll also do tests to try out all the abort modes, to make sure that the code does the right thing. We’re trying to get this done before our new engines are in so that as soon as we get them in we can move directly to qualification testing and then to actual vehicle testing.

Until next time.