CRSO Letter

3393 Eddie Road

Winnebago, Illinois 61088

January 16, 2017

CRSO EIS

P.O. Box 2870

Portland, Oregon, 97280-2870

Subject: Automatic, Unattended Type-1 Optimization is Available Now

It is in the best interest of all concerned parties that each and every Kaplan turbine on the Columbia River operates at its best possible efficiency to maximize generated power, extend operating life and minimizes environmental harm. A unique requirement for Kaplan turbines is they must be individually index tested and optimized periodically by tuning-up their 3-D Cam surfaces by what is referred to as “Type-1 optimization."

Index-testing, or Type-1 optimization is a tune-up that has four purposes:

1. To determine the overall efficiency performance envelope to provide the information necessary to maximize generated power and revenue from the available water; especially in a Joint-Load scheme in multi-unit powerplants (known as Type-2 optimization).

2. Diagnostic assessment to determine a turbine’s condition to indicate when refurbishment is needed.

3. A special case for Kaplan turbines is to optimize their 3-D Cam head and gate to blade data surface to peak the efficiency envelope across their entire operating range.

4. And most importantly, harmful environmental impact of hydropower on downstream migrant aquatic life is minimized when Kaplan turbines are operated at their best efficiencies.

The recommended interval for this tune-up is once every 5-years for each machine. For example, at Wells Dam there are 10 machines. Two are index tested every year on a rotating schedule so each and every one gets retested every 5-years. In contrast, USACE (US Army Corps of Engineers) has stated that their working standard is to index-test only one unit out of each type family over their entire operating life time. Testing only one machine is sufficient to rough-in the 3-D (three Dimensional) Cam profile for the machine family at initial startup by refining the model-test data - but this is insufficient to achieve maximum overall long-term efficiency from the entire Federal Columbia River Power System FCRPS turbine fleet.

For many years the stated goal of government hydropower engineers has been “Automated, Unattended Data Collection,” which will alleviate the high labor costs and drudgery of the field-work of index-testing.

Unattended Field Testing has been Within the Government’s Reach Twice

Within the last 30 years; on 3 separate occasions Bonneville Power Administration (BPA) has funded field-test demonstrations of a successful automatic Index Test Box (ITB) for Kaplan turbines that would solve the Gordian knot that is Type-1 Kaplan optimization in FCRPS. Unfortunately, in every case, USACE’s objective was only to learn how the new technology worked to reverse-engineer (or copy) it in order to create government-owned technology that could then be farmed-out to a local Captive Supplier.

The ITB technology was initially developed in 1985 by Woodward Governor Company¹ in Rockford, IL. When BPA purchased the first ITB, Woodward acquired a U.S. Patent² to protect their new technology.

It is important to note here that BPA purchased and demonstrated Woodward’s ITB in 1986 and again funded the purchase and demonstration of Actuation Test Equipment Company's (ATECo) ITB in 2005. BPA bought two ITBs and funded three separate field-tests for them but never got any value for the money.

1) In 1986 BPA purchased and successfully demonstrated Woodward’s ITB at PGE’s Bull Run Dam³. After this successful field-test demonstration BPA offered to underwrite this new unattended data collection capability by the deployment of Woodward’s new ITB into every USACE mainstem Columbia dam to index-test all of their Kaplan turbines on the Columbia River. This project would have included upgrades to new Woodward governors and 3-D Cams on almost every turbine. Inexplicably, this offer of FREE, successfully proven equipment from a leading private sector supplier - paid for by BPA - was declined by USACE. Fifteen years later we learned why*.

*The why is that USACE had developed their own design of a Kaplan 3-D Cam and installed them at Bonneville Dam, supplanting Woodward’s perfectly functional equipment that was previously installed on these turbines in 1980. Because of this swap compatibility to Woodward’s new ITB was no longer available for BPA’s proposed first ITB field test demonstration there⁴. This swap was not learned of until the 2^ndand 3^rdBPA funded, successful ITB demonstrations in 2005⁵ and 2006⁶. At the same time it was learned that USACE had also been working on their own automatic index testing device⁷ - back in the day.

USACE HDC has also had numerous failed development projects attempting to devise a suitable Kaplan governor and 3-D Cam at government expense. After several failed attempts to create a digital electronic Kaplan governor for their Captive Supplier to mass produce and sell back to USACE, the decision was made to buy new governors from someone who knows how to make them already, instead of creating their own design from “whole cloth.” This new equipment was purchased from the private sector because the government engineers were unable to design and build a workable governor. The same is happening with the automatic index testing device that HDC has been working on since the early 1960’s (According to Lee Sheldon; when first contacted in 1984 Lee said the government had been trying to automate index testing for 25 years already with no success, and many in government deemed index testing Kaplan turbines with a computer an impossibility). The successful field tests at Clarence Cannon Dam in Missouri⁸ and Bull Run Dam in Oregon³ showed that it was possible to index test a Kaplan turbine with a computer, so after these successful demonstrations (the first funded by Woodward Governor Company and the second funded by BPA) USACE resumed their project to develop their own “automatic index testing device.⁷”

In 2004, after having abandoned their “automatic index testing device” project and then Woodward’s U.S. Patent² on the successful ITB expired, USACE sole-sourced⁹ the acquisition of the legacy technology¹⁰ that had been previously demonstrated by a joint effort at PGE’s Bull Run Dam by BPA¹¹, Portland General Electric (PGE)³ and Woodward¹² in 1988. As a result, USACE purchased the first of ATECo’s newly resurrected and updated (ITBs)¹⁰, which was desired because it had a legacy to Woodward’s original successful ITB from 1988³. After ATECo’s ITB was demonstrated successfully at McNary Dam¹³ and Ice Harbor Dam¹⁴, USACE co-opted the project and diverted the money to an internal government project to replicate ATECo’s ITB because the government engineers chose not to acquire the genuine ITB technology from the private sector. Claims were made that the ITB software had been written at government expense and therefore must be turned over to the government. These claims however, were unfounded due to the fact that ATECo wrote the ITB software at private expense and obtained a U.S. Copyright¹⁵ with a Special License Agreement¹⁶ prior to signing the government contract¹⁰ to protect ATECo’s proprietary new technology.

The Index Test Box is primarily a software based product; without the source code it cannot be duplicated. The original ITB was protected by a U. S. Patent²; the current version of the ITB was protected by a Special License Agreement¹⁶ inserted into the government contract¹¹ that referenced the U.S. Copyright¹⁵.

At the signing of the contract, the software source code was offered at $750k¹⁷, but after the successful demonstrations of ATECo’s ITB, USACE engineers opted not to buy the software and started working to make their own ITB design.

When Woodward’s Hydro Division failed in 2002 the ITB project was abandoned, and then subsequently resurrected by the original Inventor into his Actuation Test Equipment Company (ATECo).

2) In 2004 BPA Hydro Optimization Team (HOT) funded a sole-sourced⁹* USACE Contract for a Type-1 Optimizer** that was to have a legacy to the original successful device demonstrated in 1986¹¹. The initial demonstration⁵ at McNary was almost completely successful; unfortunately there was an unexpected problem with the hydroelectric unit. ***

* The ITB was sole-sourced⁹ because USACE wanted to assure a legacy to Woodward’s successful technology, no substitutes would be accepted.

**Type 1 optimizer includes index-testing for Kaplan gate to blade cam tune-up.

***There was excessively high noise on the Winter Kennedy (W-K) taps on Unit 9 at McNary Dam, the unit designated for the demonstration. It was explained that the problem was that the W-K taps were not in the same radial plane as prescribed in Ireal A. Winter’s design¹⁸. The ITB program was modified at no charge to accommodate the high noise levels.

3) After a few ITB software bugs found at McNary were fixed, a second demonstration at Ice Harbor was completely successful, as shown by ATECo’s data analysis¹⁹,* HDC Test Engineer’s email to the project Technical Lead²⁰ reporting informally on the field test and the subsequent PowerPoint presentation made back to the HOT²¹ to get more funding for the next phase of the project.

*This is the first time the ITB was used to analyze pre-recorded data, where a continuous data recorder was streaming data to memory while the Kaplan turbine’s gates and blades were positioned manually at the positions required for a 3-D Cam index test and optimization (or tune-up) and then the data was sent off-site for subsequent analysis using the ITB. This method has been improved and expanded for the current ITB used to index-test the Kaplan at Dorena Dam, as described below.

Bait and Switch

After the second demonstration at Ice Harbor Dam USACE told ATECo that the test was successful and that the government wanted to buy two more ITBs, one with GDACS (Generic Data Acquisition System) interface and the other with discreet transducers and sensors, however the Purchase Order would have to wait a month because of HOT meeting scheduling - but ATECo should get started right away building them because when the BPA HOT released the funds to buy them USACE HDC wanted to move on this right away.

Shortly thereafter, it was learned from an inside source that while telling ATECo that the government wants “two more right away,” the HOT Chairman was circulating an email²² containing a PowerPoint that claimed the “ITB source code was running on the GDACS platform²³” (Slide# 14²⁴), which was an incorrect statement making an untrue claim that the ITB source code was already in the government’s possession. (The price for the software source code was set at $750k. The money was never paid, so the software source code was never given to the government.)

Subsequent FOIA responses showed that the HOT Chairman and numerous USACE personnel were claiming that the ITB technology had been developed by government engineers instead of hacked from a private sector vendor. An email from the USACE Contracting Officer’s Representative²⁵ (COR) acquired via a FOIA request years later pointed out the conflict of interest within HDC over this project.

A subsequent conversation with the head of Oak Ridge National Laboratory (ORNL) Wind and Waterpower Group in 2010 about a potential ITB demonstration for an upcoming project was derailed by HDC personnel. ORNL suddenly broke off contact with ATECo, and when pressed said that USACE personnel said that whatever ATECo was offering had been developed by HDC as government technology under a prior contract. To back it up, ORNL provided a copy of the job posting²⁶ received from HDC that stated the ITB was developed by the government on a “prior contract.” When this was checked out, a frustrated FOIA officer said the only prior contract was with ATECo²⁷, so the prior contract mentioned in the job posting was actually ATECo’s, but that’s not what ORNL was told by HDC.

The significance of this is that as government property, the ITB could be given to a favored Captive Supplier²⁸ to mass produce and sell back to the government and elsewhere instead of truthfully stating that the ITB technology was proprietary to ATECo, and that in only one instance had an ITB been purchased by USACE from the original inventor.² - just to see how it works so USACE personnel and their Captive Supplier could duplicate it with a DOE funded government project.

When it was subsequently learned by ATECo that the funds in DOE’s ITB contract earmarked to buy up to 320 of ATECo’s ITBs (after the successful demonstration) were being diverted to HDC’s Captive Supplier to fund a project to reverse-engineer ATECo’s ITB, complaints were made to DOE IG (Department of Energy Inspector General) about mismanagement of funds. DOE IG and OMB said that contract funds earmarked to buy equipment from one supplier could not be summarily diverted to another vendor²⁸ to copy the first vendor’s product. The HOT Chairman sent the email advising the other involved parties about this prohibition. It was later learned that instead of stopping this chicanery, the three individuals working on the ITB project at the Captive Supplier were retirees from HDC; so they could leave their jobs at the Captive Supplier to be taken back in at HDC as “rehired annuitants” to work on the same project to reverse-engineer ATECo’s ITB while “in-house” at HDC. This unethical, but legal practice continued until the ITB contract money was exhausted, and then the project name was changed³⁰ to Gate Blade Optimizer (GBO) to obfuscate what had happened and find new sources of revenue in a classic “Bait and Switch” maneuver.

If CRSO and the listed government agencies want to increase FCRPS overall operating efficiency, they should re-examine the previously purchased and successfully demonstrated ITB technology for Kaplan turbine Type-1 optimization that BPA has already paid for, twice, but never got any value for the money. The reason for this is because the government was scammed by the government personnel’s scheme to co-opt the ITB technology into an in-house project for themselves and a favored Captive Supplier to profit from.

Contract Point that got Lost in the Shuffle

At the onset of the ITB project in 2004 the ITB program software source code was not included in the deal. This Intellectual Property was treated like Microsoft’s Windows program wherein the end-user just licenses one executable copy of the program software to use and does not get the right or ability to duplicate and distribute the program – which is what the Government engineers really wanted all along.

As the final contract negotiating point in 2004, USACE demanded a side-deal for a firm, fixed-price for the software source code so it could be incorporated by government personnel into their GDACS program should the ITB “proof of concept” demonstration prove successful. A price of $750k was arrived at and agreed to¹⁷ with the USACE Contracting Officer’s Representative (COR).

After the successful field test demonstration a USACE engineer (who had never actually written any software himself) decided that writing the ITB software was “no big deal,” and that the government didn’t need to buy the successful ITB source code from the supplier in order to duplicate the ITB - the Captive Supplier could duplicate it with ease. This engineer pushed for the ITB project to be canceled and the funds transferred to an in-house USACE project for himself to duplicate ATECo’s ITB, as noted in an email from the COR to the project Technical Lead³¹.

As one might expect, the project and money went their way. Subsequent FOIA responses over the last 10 years have documented failure after failure of various government engineer’s projects attempting to duplicate the genuine, original Index Test Box’s demonstrated capability of Unattended Data Collection, first with a government project that was still named “Index Test Box” to make it seem it was the same project named in the contract, and then after the contract funds ran out and the IGs started taking note, the name was changed to “Gate Blade Optimizer (GBO).

The net result of all of this is Woodward’s (and then ATECo’s) successful ITB work with BPA has been perverted into a $1M/year boondoggle for government personnel, when it should have already evolved into a comprehensive index testing and optimization program for all of the Columbia River Kaplan turbines years ago.

====================================================

Well, it’s been 10 years – let’s compare results.

USACE’s Gate Blade Optimizer (GBO) (formerly known as Index Test Box or, ITB) project has yet to index-test and tune-up a single Kaplan 3-D Cam surface after 10+ years. Untold millions have been spent trying to replicate proven private sector technology while earlier this year ATECo’s ITB was successfully demonstrated thrice at Dorena Dam near Cottage Grove, Oregon - about 2-hours south of Portland.

Index Testing At Dorena Dam Using ITB and Hybrid Index Testing Method

Dorena Dam is a USACE flood control facility that was retro-fit recently with two generators (6.1MW Kaplan and 1.4MW Francis) as an investment by a private firm in Toronto, Canada to capitalize on the free energy available from this USACE flood control dam in Oregon.

After the units were commissioned, the Kaplan was not producing as much electricity as anticipated. Consultation with the turbine vendor indicated a full 4-head optimization needed to be conducted to tune-up the 3-D Cam in the Kaplan blade controller for maximum efficiency under all operating conditions.

Lee Sheldon was engaged to index test the Kaplan unit, but his teaching schedule and medical appointments precluded his going to the dam on short notice for 4 separate conventional index tests, each taking 3-4 days to conduct so another plan was needed.

Lee engaged ATECo to utilize the ITB to take advantage of its Remote Index Testing capability that was first demonstrated at Ice Harbor in 2006 and again in 2010 on the 3 Kaplan bulb units at the Clergue powerplant in Sault Ste. Marie.

The Index Testing was actually conducted by Northwest Engineering Services (NES) and HydroTech Engineering Services (HTES). These guys renamed the index-testing process “Hybrid Index Testing” for their project.

The gist of the Hybrid Index Testing Method is to permanently setup the existing SCADA system instrumentation and data recorder for an index test and then leaving the data logger running continuously, recording scans at 2 Hz. while the gates and blades are positioned at the various combinations needed for an index test. A scan includes time, date, head, tail, gate, blade, flow power, net head, power factor, frequency etc.

These “canned” data files are then transferred one-way to an off-site location using CDs, thumb-drives or other physical memory or by emailing the data from a terminal not connected to the SCADA system, thus avoiding any powerplant security issues. When received at ATECo, the data is analyzed and reduced to a new 3-D Cam surface to install in the governor’s 3-D Cam blade controller.

Data analysis consists of playing back the continuous data streams and analyzing for steady-state operation using the ITB. This operation is just as if the ITB was connected to the unit. The difference is the ITB can analyze the data at up to 100x normal speed. Weeks of data can be analyzed in a few hours which makes the process more convenient, timely and at a much lower cost.

Once the steady-state data is gleaned from the continuous recordings, the resulting SteadyState data is sorted by hand to get the best representation of the desired data set.

With the Hybrid Index Testing Method there are No Security Issues, Ever

Anyone who has ever worked in a federal powerplant knows that it often takes longer to get the necessary security clearance to enter the dam than it does to do the work. This fact has discouraged index-testing on many occasions, but this need not be the case anymore. The security aspect of index testing is completely avoided by the Hybrid Index Testing Method because the index-test engineers don’t need to go into the dam. All on-site work is done by normal facility personnel. No outside index test engineers are needed and data transfer is all one-way using thumb-drives or internet data transfers so there is no chance for malware or viruses to ever migrate into the SCADA system via the index-testing process.

Dorena Index Test Setup

For this index test the normal site maintenance crew personnel at Dorena setup the SCADA instrumentation as typical scheduled work-order tasks. The index-test was then executed in a few hours by NES and HTES (the engineering and maintenance team hired by Dorena LLC to oversee and perform operations and maintenance tasks at the dam).

To run the index test NES and HTES exercised the turbine gates and blades and collected the data in about 3-hours. The data then was sent to ATECo for reduction. The reduced index test data was then sent to Lee Sheldon for final analysis. Test reports documenting the 3 cam-profiles for 48, 69 and 85 ft. head are available. There’s still one more index test left to do before the cam profile is complete. We’re waiting for the water level to come back up this spring, perhaps it will get high enough this time...

Why the ITB Works

The key to “Unattended and Automated data collection” is a software algorithm that detects and gleans “steady-state” operating points from the continuous data streams and noise emanating from the SCADA system on a hydroelectric unit. This software algorithm originated in several automated aircraft gas-turbine engine fuel control test systems developed by the inventor at Woodward Governor. A technology transfer from the Aircraft Division to the Hydro Division resulted in Woodward’s ITB. Anyone who has ever personally worked at index testing knows that getting truly steady-state data points every time is exceedingly difficult and time-consuming, and doing the manual process 50 or more times in quick succession over several days for an index-test gets very old, very fast.

Hydro units are very noisy machines operating in a very noisy environment. From time to time, a hydro unit and its environment will settle-down and everything becomes very quiet. The ITB SteadyState algorithm will automatically detect these “quiet-times” and record steady-state data points whenever the unit is seen to be running steady-state.

ITB Reduction is better than Conventional Hand Reduced Data

It has been stated that the resolution and accuracy of the ITB results provide greater coherence to the “smooth-curves” for the new 3-D Cam surface than conventional manual index testing.* The resolution and coherence are so fine that observation can be made of the changing energy levels in the water caused by starting and stopping adjacent units**, indicating an improved Kaplan control methodology is possible.

*Lee Sheldon made this observation for the Bull Run Dam test in 1988; the Ice Harbor Dam data in 2006, and most recently for the Dorena Dam test last year (2016).

**A New Kaplan Control Model is indicated

This data indicates that Kaplan blades would be better controlled by a flow to blade angle 2-D Cam instead of a head and gate opening to blade angle 3-D Cam.

The reason Kaplan blade controls are the way they are now is that when the Kaplan turbine was invented in 1913 by Professor Kaplan, flow was much more difficult to measure than it is today so his original 3-D Cam control algorithm used head and gate to predict flow instead of measuring it more directly.

Today low-cost differential pressure transducers across the Winter Kennedy taps and a flow to blade-angle 2-D Cam would make a much better control system. Measuring flow directly is insensitive to the variations caused by starting and stopping adjacent units, unlike predicting flow from head and gate opening.

This simpler and more elegant control algorithm would not work during power-level transition so using the original head and gate to blade method during power level changes will provide good dynamic response and stability during these transitions. After things settle out, the flow to blade angle cam can then take over to trim the blades for long-term efficiency maximization.

For any benefit to occur, individual turbine data from recent field tests on each and every turbine must be utilized (or, the 3-D Cam surface must actually be updated with the new data for any real benefit to occur).

A fundamental factor of hydropower optimization is that identical hydro units will not achieve the same operating efficiency under identical conditions.

This was demonstrated shortly after the first batch of turbines at The Dalles were placed in service in 1960³². Efficiency performance of all of these machines was measured individually using absolute-value current-meter index tests.

Despite the fact that these turbines were as identical as humanly possible - all of the same size and design from the same vendor purchased on the same contract, the absolute-value current meter efficiency data showed that peak efficiency operating points varied over a range of 9% while the power levels of these peaks were spread over a span of several MW.

The takeaway from these tests was a strong argument for index-testing each and every unit and tuning-up their 3-D Cams individually by developing a unique 3-D Cam data surface using recently-measured real world index-test field data. USACE has yet to achieve this goal; despite this known reality, the common practice within USACE powerplants is to use the same 3-D Cam profile in all machines of the same type family as If they are all identical - which is known to be untrue.

This “short-cut” results in millions of dollars of lost revenue annually.

Costs Involved

A typical index-test at a single head followed by a 3-D Cam optimization consists of efficiency measurements at various gate to blade combinations to define the new best-cam line for the existing head and then morphing the entire 3-D Cam surface about the new best-cam information for the existing head. This test requires the logistics of travel, lodging, meals and labor for 2-3 days of setup, field-measurements and teardown.

Likely reasons this index-testing and optimization is not getting done are because it is difficult, tedious, labor intensive, boring and expensive. In 2005, the government’s budgetary figures for the conventional index tests of the Kaplan turbines at McNary and Ice Harbor by HDC was $25k (over $30k today using government CPI figures), and that the fun of making week-long field-trips to the dams wears thin very quickly, all of which definitely would tend to spur-on efforts to automate the field-data collection part of index-testing.

USACE’s logical desire to comprehensively index-test all of the turbines was demonstrated by creation of a tutorial manual detailing how to field-test a Kaplan turbine. Due to its red cover it was referred to as the “Red Book.³³”

Apparently after the thrill of going on 3-4 day field-trips to the dams every week wore off this field-work effort was discontinued. In the 1980's, a large cardboard box of these manuals were given to Lee Sheldon for disposal because they weren’t needed anymore, perhaps because the automated system to alleviate the drudgery of taking field measurements was promised to be, “just around the corner...”

It’s still not ready.

In closing, my point is that a successfully demonstrated and proven Type-1 optimizer has been available for over 32 years from the private sector, been purchased twice and evaluated by BPA thrice, yet government personnel still continue to spend untold millions of dollars unsuccessfully trying to duplicate this proven, private-sector technology as a government project. One must ask: is it vanity, or just greed that is driving this misdirected effort?

Sincerely,

Douglas J. Albright, President

Actuation Test Equipment Company

Linked references, in order of appearance:

1. http://actuationtestequipment.com/Reference_Materials/1987-07_Hydro_Review.pdf

2. http://actuationtestequipment.com/Reference_Materials/1988-12-27_U_S_Patent_4,794,544.pdf

3. http://actuationtestequipment.com/Reference_Materials/1987-09_to_89-02_Gary_Hackett_Report.pdf

4. http://actuationtestequipment.com/Reference_Materials/1986-06-11_BPA_to_Corps.pdf

5. http://actuationtestequipment.com/Reference_Materials/2005-12-12_Rod_Wittinger_ITB_T1_Trip_Report.pdf

6. http://actuationtestequipment.com/Reference_Materials/2006-02-21_Dan_Ramirez_Ice_Harbor_Report.pdf

7. http://actuationtestequipment.com/Reference_Materials/1990-03-27_Sheldon_Evaluation_of_Thorson_ITB.pdf

8. http://actuationtestequipment.com/Reference_Materials/1986-04-16_Sheldon_on_CC.pdf