Dear Mr. Allen,

I was already aware that the data that I have now is all there is to be had from the ITB testing. What I was inquiring of Lee and Dan was a fuller understanding of the extent and nature of the USACE turbine index testing that had been done since 1999, it was not a request looking for more ITB testing data. However, I would like a copy of the final Ice Harbor Unit 3 test. I needed to know the scope of your own index-testing program so I could formulate the following arguments.

The data presented herein indicates that HDC has been covering up a big problem for years, and I am attempting to uncover it so it will finally get fixed. I believe the truth will win out. I didn’t mean to slight you personally; I just needed a truthful answer. It’s not about you, USACE or me, it’s about the truth, and getting the job done right.

I believe that USACE, as custodians of a critical national infrastructure has obligations to maximize revenue to the Treasury for the taxpayers, to minimize fish mortality for the Indian tribes and future generations of Americans by fixing these problems now, instead of preserving the status quo and furthering the economic fortunes of HDC’s favored supplier, ACSI.

You may or may not be aware of it, but the turbine-testing program currently in place by USACE is terribly deficient. Some author wrote a letter to Congressman Greg Walden in response to my initial diatribe on the subject, touting 40 (or so) index tests on 20 units in 7 years as a significant step towards fixing the problem.

This is a laugh - this pace has only tested about one turbine per powerhouse in the last seven years - this is altogether inadequate. A proper test program would be testing all of the turbines in every powerhouse at least once every 5 years. A more modest, yet still acceptable interval would be every ten years, but my information shows that most of the turbines on the Snake and Columbia Rivers have never been index tested.

Furthermore, even if the turbines are index tested, the information from these tests is not used to update the cam surfaces in the individual units to improve each one’s operating efficiency. We know that all turbines of the same family in a given powerhouse are operating on the same cam profile, an altogether unacceptable situation.

To get the job done right, the pace of your index testing should be more on the order of 46 index tests per year to comprehensively test these units, both with and without screens.

And while I’m being critical, your control systems could use some work too. The one-degree deadband and 35-second deadtime programmed in the 3-D cam blade control systems that I witnessed, and Rod Wittinger and Dan Ramirez documented in their ITB Field Test at McNary in December 2005 are also a big problem. These deficiencies are meant to mask other control system instabilities that make the units continuously hunt and wander during normal operation, prematurely wearing out their blade mechanisms.

Just so we both are on the same page on this; in August of 2004 while I was there for a preliminary site survey to look at your unit #5 at McNary, after I explained how the ITB worked by testing along the Constant Power lines of the turbine’s operating envelope, Dan Perrier explained the nature and mechanism of the unit instability to me - and why my plans wouldn’t work.

Dan explained this behavior as if it were normal for all turbine governor systems to work this way; it seemed that he saw nothing wrong with this situation. He was trying to discourage me from attempting to use my ITB to delineate the optimum cam surface using the Constant Power method of testing. My observations of the unit’s behavior, and the data captured by the ITB on Unit 9 at McNary matched his explanation exactly.

It goes like this:

1.) When a large upward SetPoint change is made in power; Starting at (a.) on figure 1, the speed motor is activated to raise the SetPoint by the GDACS control system, and the gates open rapidly to get the power output to the new SetPoint level, as shown by constant power level line (b).

a.

c.

b.

Figure 1 Rectangular Spiral motion of GDACS 3-D cam.

The increase in power output is sensed by the GDACS control system, which continues to open the gates until the generation SetPoint demand is satisfied at (b).

Due to the 35-second deadtime in the blade control system, the blades move only in response to the linear rise cam mounted on the restoring shaft, indicated by the line from (a) to (b). During this time, the stepper motor that provides the rest of the required motion does not move immediately. This causes the gates to move farther than they would have had the blade controller been robust and accurate and the blade angle tracked the cam curve.

About 35 seconds later, the blade controller wakes up, takes note of the change in gate position, computes the new Ideal Blade angle, and then moves the stepping motor to complete the blade motion to the new Ideal Blade position for 75 ft head at (c.), as dictated by the 3-D cam algorithm and cam surface lookup table.

When the blades move to this new, steeper position, power output increases significantly, exceeding the deadband set for the power level controller in the GDACS, so the closed loop on power in the GDACS pulls the gates back closed to get power back to the SetPoint.

After another deadtime in the GDACS load feedback, the speed motor is moved again to lower power back down to the load SetPoint at (d.). Again, the blades do not move in unison with the gates to track the cam line because the 35-second deadtime is in effect again.

This motion continues until the blades are within the 1-degree deadband programmed into the Panel Mate touch screen control panel.

To avoid this “rectangular spiral,” (or to better characterize the impact this misalignment would have on any fish passing through the turbine during this sequence, we could call it a “rectangular death spiral”) the operators change the generation SetPoint very slowly so the blade control system can keep up.

In addition to this problem, there is a 1-degree deadband that prohibits blade motion if the blades are within 1 degree of the Ideal Blade position, resulting in a permanent and continuous 6.5% blade position error away from the Ideal Blade position.

As defined by ASME PTC-29 measurement technique, which sums the deadband on both sides of the optimum cam line, this is a 13-degree deadband. Add this to the anticipated 2% deadband in the mechanical hysteresis of the blade positioning system of the machine down stream of the oil head transducer, and they are now operating with a 15% blade positioning deadband.

IEEE Std-125 characterizes an acceptable deadband at 1%, which all other suppliers conform to. This is not a mandate, however, these are just guidelines indicating what is reasonable and customary throughout the industry for contracting agents to refer to in preparing purchasing documents.

If other suppliers can achieve this level of accuracy, why is USACE accepting such an inaccurate system from ACSI?

The consequence of this misalignment is to exacerbate the fish mortality problem that the Environmentalists and a Federal Judge are so upset with USACE about.

I had a professor in college who described business relationships such as the one between HDC and ACSI as “industrial incest.” Not a very flattering name, but descriptive of what I see going on.

It is incumbent upon USACE HDC to deploy the best possible systems to prevent such problems, and to not allow any industrial incest or favoritism to get in the way of doing the job properly.

The data from the index test of Unit 9 at McNary is presented in figures 2 & 3 to show the cost in lost efficiency, (aka lost power and/or aka lost money) resulting from this continuous gate to blade misalignment so the losses to the taxpayers can be estimated.

Figure 2 McNary Unit 9 data.

This is the ITB Field Test data from Unit 9 at McNary, run in December 2005. Note the green arcs cresting over at the top are the relative efficiency of the unit. The reason they change color to aqua on the right side is the head & gate to blade alignment in the GDACS 3-D cam is so far off that the actual optimum cam line is so far away from the cam line programmed into the GDACS 3-D cam that even with the 2-degree authority granted to the ITB blade perturbation routine, and tightening up the deadband to 0.5 degree, we still couldn’t reach to the peaks and over the crest of the efficiency curve to the decrease on the other side. The misalignment problem is quite large as indicated by this data.

The HDC ITB project team allowed me to extrapolate this data to show where the crest might be so I could demonstrate the new optimum cam definition procedure envisioned for ITB constant power data, as shown I figure 2. I was conservative in this estimate to be kind, but let’s be a little more realistic now, continuing with the analysis below.

In March 2006, seeing how HDC was handling this problem; instead of using the ITB that was proven to work at both McNary and Ice Harbor to identify and fix this problem ASAP - a “business decision” was made that my ITB was incompatible, and the same company that made a mess of the 3-D cams, as described elsewhere in this document would also be developing a knock-off of my system under the auspices of a dubious “in-house/hired contractor” arrangement to make a “generic ITB via a method analogous to GDACS.” It’s too late for this kind of foolishness; you guys need to fix this now.

Adding some graphic analysis to figure 3, the efficiency loss resulting from a 1.0 degree deadband error is estimated, as well as showing how far off the 3-D cam surface currently installed in the machine is.

Point “a” is the Ideal Cam position from the GDACS 3-D cam running at 53MW at 75-foot head. Assuming this is the actual efficiency peak of the machine (it’s not), project a line up one degree’s distance to point “b,” to show a 1-degree steeper blade angle - the deadband error programmed into all of the 3-D cams at McNary.

Now, assuming we’re operating at the same power level as point “a,” project a horizontal line over to intersect with the 53MW Constant Power line at “c” to find the new gate-blade operating point.

Now project two vertical lines up to meet the efficiency contour at “d” & “e.” The difference in elevation between “d” & ”e” is the lost efficiency from operating at the same power level with a 1-degree blade error, if the Ideal Blade position at “a” was the optimum point.

But it isn’t; the actual efficiency peak indicated by the index test data we collected is somewhere around “f,” the extrapolated point that is projected beyond our test range. The lost efficiency is actually the vertical distance from “d” to “f.”

Using the same graphical method, the distance from “d” to “f” is measured to be a loss of 1.5% efficiency.

This analysis was done along the 53MW Constant power line of the unit, so the 1.5% loss is calculated at 53MW. The math is simple:

Assume 5 cents/kwh at the meter on the side of the house.

(I’m ignoring distribution costs and other overheads because they are already paid for out of the diminished power level that is already being obtained from the machine. By just moving the blades to the proper angle, all of the power to be gained is pure profit.)

Assume that the unit is achieving 85% efficiency at 53 MW when we start, and the 1.5% is added to increase efficiency to 86.5%.

The 1.5% increase was the wasted portion of the total power in the water, so if 53MW is 85% of the power in the water, then 1.5% more is = .935% of the available power in the water for purposes of determining compliance the 1% edict. A very slim operating margin is left to run this machine during the restricted operation period.

The power output increase computed as:

If 85%=53 MW, then 86.5% = 53.94MW, an increase of .935MW.

The retail value of this is .935MW * 1000kW/MW * .05/kWh = $46.75/hour.

Assume 50% running time year round, the annual gain is $204,765/year.

This is the annual cost of the sub-standard 3-D cam and blade control system on unit 9 at McNary. I would be willing to bet the losses on other units in that powerplant are comparable.

I’m told that the instability problem is often so bad, the operators shut the 3-D cams off so the blade position input from the SoftPLC computer and stepping motor does not update with gate movements. The result is a blade motion that tracks only the linear-rise cam for the blade controls system, causing additional efficiency losses. This is shown in the data graph of Fig 4, which was captured by the ITB during an overnight run on Unit 5 at McNary on September 8, 2005. The operators forget to turn the blade controller back on when they changed the power level, so the actual blade motion across this wide operating range has the added head & gate to blade error of not having the stepping motor input in the actual blade position at all.

When we were at McNary again in December, working with Unit 9, when I asked, Rod told me not to leave the ITB running overnight to test for problems with system crashes etc - he said no, he was told not to allow any overnight running of the ITB while we were at McNary. My guess is someone didn’t want me to see something like this happen again.

I have been in your McNary powerplant and seen these problem up close and personal. Data from Ice Harbor indicates similar losses due to misalignment of the 3-D cam’s head & gate to blade surface.

From the data I received under FOIA request recently, it appears there was a calibration difference between the original data I received in March and the data in the FOIA package. Either way, the optimum cam line from the data appears to be below the surface installed in the 3-D cam by a similar error to that seen on Unit 9 at McNary.

From conversations with your people, I know the problems are systemic throughout your powerplants. The bureaucracy within HDC has compartmentalized the responsibility and authority for determining operating methods and parameters of these machines to such a degree and in such a manner that it is not conducive to proper deployment and support of the control systems for the turbines. This needs to change.

These problems exist to the determent of the revenue stream going into the National Treasury and a worsening of the environment for the fish that have become the focal point of much controversy.

It is obvious that internal politics and power struggles within the Corps have pre-empted the responsible and proper care of these machines, and also derailed my project in favor of having ACSI make their own ITB to be used in all Corps plants and marketed beyond USACE’s powerplants into the commercial market.

My diatribes to your masters of late are an attempt to disrupt the current bureaucratic logjam within HDC so a more proper alignment of responsibility and power that will be more conducive to getting the job done properly, can be arranged. As the saying goes, you’ve got to break a few eggs…

I’m not doing this as a contractor in an attempt to get the work for myself; I don’t want or need your business. I’m doing this as an American Citizen who sees a problem, and can also see a solution, and I also see how the desire to preserve the status quo at HDC is preventing anyone within your organization who knows that these things I am saying are true from doing anything about it for fear of getting fired.

My original ITB was offered to USACE by BPA in the late 80’s, to be paid for with DOE funds to fix this problem back then. New Woodward Governor electronic 3-D cams would have also been required for all of these units, making a tidy system that would have taken care of all of these problems very nicely - with a world-class organization behind it to continue development and support. But USACE didn’t want this proven system; you decided to make your own 3-D cams and index test boxes.

In the words of TV’s Dr. Phil, “…and how’s that workin out for ya?”

The driving force behind part of the fish mortality problem started with USACE’s earlier installation of the Seawell blade controllers many years ago, these didn’t work at all. Recently these were just swapped out for the current GDACS 3-D cam system at McNary, all since the year 2000.

It must be acknowledged that the GDACS 3-D cams from ACSI are an improvement over the Seawells, but neither work adequately well compared to other offerings from many competent, reputable vendors today.

If USACE had jus allowed BPA to provide this equipment from Woodard, GE, Voith or any other of the reputable suppliers offering such systems, the problems described above would not have become so bad as to require the Federal Court’s intervention.

The mechanical engineers within your organization who are familiar with turbine design and operation should be dictating the requirements of proper turbine control parameters to the GDACS control system computer people, not the other way around - which unfortunately is the way it is.

When I went to McNary with Rod Wittinger, I perceived that he had been excluded from the design and installation of the GDACS 3-D cam control system there. He did not know what configuration of control hardware was on the units when we arrived there, and I could tell that he was dismayed that the installation procedure for the new GDACS 3-D cams did not capture the calibration and alignment of the gate restoring system of the unit when the new system was installed, thus rendering all previous index test data for Unit 5 useless to our Index Test Box proof of concept testing. He tried to bluff me as to the extent of the problem – I want to play some poker with that guy some day…

When we started working on the unit, and whenever we asked, nobody really knew where the gate calibration was in relationship to the blade calibration for the 3-D cam setup. This is altogether unacceptable.

For whatever self-serving reasons, the control system built by the favored “In house/hired subcontractor” arrangement, and however well it performs, are as good as it's going to get, regardless of how much better other commercially available equipments from others may perform unless some pressure from outside the Corps is applied.

If the current GDACS control system supplier cannot meet the nominal industry standards presented in IEEE Std-125 as tested and verified by PCT-29 like every other reputable vendor does, then they should not have a monopoly on USACE’s business of providing control systems for these turbines.

Every other reputable control system vendor in the business meets these specs, and if USACE bought from them, I’m sure they would be taken to task if their equipment were sub-standard.

I have seen that the incompetence and system problems I observed while within your facilities are being hidden behind national security guidelines. That's not what national security is all about, Sir.

I'm not the enemy.

I'm an American citizen, a Veteran, and while I served in the USAF I had a top secret clearance that enabled me to work in the aircraft control tower and in the Tactical Fighter Squadron's Command Post and runway observation unit (ROU) while combat aircraft were flying at McCord AFB in Tacoma Washington, in the SAC Command Post and SAC HF Radio Relay site on Okinawa Japan during the Vietnam conflict, where they were passing top-secret radio traffic to the B-52 bombers, KC-135's and SR-71 spy planes flying in support of that mission, and at SAC's Hawes Radio Relay station at Edwards AFB while they were passing top-secret radio traffic to submarines lying on the ocean floor. If they could trust me, you can trust me.

To have "anti-terrorist guidelines” used to prevent me from seeing the software source code that is at the heart of the sub-standard Kaplan turbine blade controls you have in your governor systems is an insult and an outrage.

When I needed the OPC tag names to enable me to get the necessary data from the GDACS network, I was prevented from getting this information under these USACE security policies, but I was able to purchase them directly from ACSI easily; so much for your security. It turned out to be just a hurdle that made my work more difficult, and made an easy $1,000 income for ACSI.

When I learned from your people early in 2006 that instead of continuing with my ITB, a specification was being wrapped around it so your more favored supplier could make a knock-off of it instead of continuing with the work outlined in my contract I inquired further.

From what information was gathered, I surmised that this specification was to become a “sole-source solicitation” to the same “in house/hired contractor”’ arrangement that made the mess out of the Kaplan turbine 3-D cams that are in place now. It doesn’t take a rocket scientist to figure out what’s going on out there. This is illegal, in my opinion.

From the negotiations to sell my PT Interface hydroelectric turbine speed sensor modules to your organization for the Digital Governor project, I learned that this same “in house/hired contractor” arrangement has failed to provide a workable digital governor out of three attempts to do so at Albany Falls.

Fortunately, the unworkability of this system was recognized early on - and the Digital Governor project was canceled before too much more damage could be done.

Though I don’t want the work myself due to the low prices Dave Ebner beat me down to in our final round of contract negotiations, in good conscience I cannot walk away from this and allow this same incestuous tryst that made the 3-D cams take over this project and make an even greater mess out of these turbine control systems to the detriment of the National Treasury and the Endangered Species currently being struggled over.

Instead of trying to get the ITB project away from me to give it to ACSI to make a knock-off, HDC should be compelling them to fix the 3-D cam blade control systems that are so problematic now as the next step. But USACE HDE personnel are covering up these problems on a regular basis, a practice I cannot abide.

I worked at Woodward Governor, on Woodward governors, and I know how governors are supposed to work. What you have in place now would never have made it off the shop floor to the shipping dock at Woodward; it would go out the back door into the scrap bin.

On the subject of my ITB being incompatible – I’ve been told that my ITB is incompatible with GDACS because I wrote it in Visual Basic; it is disingenuous to say it is because the SoftPLC computers are programmed in C++. That fact is irrelevant.

My Visual Basic turbine index testing program would not logically reside in the SoftPLCs anyway, it would go into a central computer located in the control room, and communicate with the SoftPLCs on the individual units over the fiber optic Ethernet.

I spoke to Cindy Hollenbeck of SoftPLC Company, who is intimately familiar with the entire GDACS control system hardware you have. She told me that the 100MBps Fiber Optic Ethernet communication system is only slightly burdened by the system you are running now, and it would easily handle the added traffic of the Index Test Box communications with the individual turbine control computers during an index test.

Let me refer you to the ACSI web page that describes the GDACS control room suite.

Click Here for ACSI Gneric DACS web page

(Or http://www.automation-software.com/Generic%20DACS.htm)

Note that on this web page they brag on the power and flexibility of the GDACS Master Station Applications, stating specifically that it supports Visual Basic applications.

On that web page they state that the GDACS control room system is Windows Based, and supports Visual Basic application programs. The configuration I would suggest is to place the ITB program into one of the control room computers, and using OPC tag names access to all of the units in the powerplant for index testing.

My program, as written, could run on this platform, accessing the necessary forebay, tailwater, gate, blade and power signals from OPC data tags from the unit under test over the Ethernet communication network. What’s incompatible about that?

The flow signal can come from any type of flow measurement system you have. I can easily adapt the ITB to intake any type of flow data signals. Analog voltage signals from differential transducers on Winter Kennedy taps, or these same signals digitized by your GDACS local to the units being tests could simply be read as another OPC tag into the ITB program. What’s incompatible about that?

Data in engineering units can be input from other supplier’s flow measuring systems such as ASL AQFlow’s ASFM using OPC tags, RS-232, RS485 etc. Any communication configuration modification changes are easily within the purview of ATECo for the ITB.

From conversations with ACSI & HDC Personnel and web browsing I’ve discerned that the ITB is absolutely compatible with every reasonable criteria, except that it is not built by ACSI, the favored contractor for all USACE turbine control systems. I challenge this level of favoritism as being illegal.

Sincerely,

Doug Albright

Actuation Test Equipment Company

______________________________________________
From:   Allen, Charlie NWP 
Sent:   Friday, October 06, 2006 3:43 PM
To:     'DudleyDevices@aol.com '
Cc:     Ramirez, Dan E NWP; Sheldon, Lee H NWP; Turner, Robert C NWP; Breiling, John J NWP
Subject:        Index Tests at COE Powerhouses

Mr. Albright,

This e-mail responds to your verbal requests of 02 October to Mr. Dan Ramirez and to Mr. Lee Sheldon, asking for information regarding the number of index tests performed by the Corps in the recent past.

Of the index tests performed by the Corps in the past 15 or so years, only four attempted tests have addressed your contract with the Portland District.    In these tests the Index Test Box was used, or attempted to be used, in all or part of the tests.  These four attempted tests occurred at:

McNary Unit 5:  The test returned no usable data because the test set-up was incorrectly performed. 
McNary Unit 9:  This test was performed and you have already recieved all data collected by the ITB, trip reports, findings, memoranda, etc.

Ice Harbor Unit 3:  This test was conducted using COE data acquisition equipment and conventional index methods. The test report documenting COE measurements and results has not been completed.  The ITB was connected in parallel to COE data acquisition equipment during testing.  All data collected by the Index Test Box has been previously provided to you.  

Ice Harbor Unit 6:  This attempted test did not proceed due to the discovery of a loss of signal between the Index Test Box and the transducer.  Because this test was being performed out of a matter of convenience, and not necessity, no attempt will be made to repeat the test.

As can be seen, you have all the vetted and finalized data that is currently available for the turbine tests that have bearing to your firm's performance under your contract with the Portland District.

If you have any future communication regarding the technical aspects of your Index Test Box please contract me at 503-808-4296 or at the return e-mail address indicated above.

Sincerely,

Charlie Allen
Product Coordinator
Hydroelectric Design Center