Archive for November, 2019

Why my first simulation included the use of a shovel

Saturday, November 9th, 2019

Early in my Superior Steel days, I was given the job of replacing one of Mr. Pete’s old relay-based control systems with a new Texas Instruments 5TI Programmable Controller. (Mr. Pete was Wendall Kitchens’ dad, and the founder of Superior Steel. He built all of the electrical controls, and his retirement was one of the factors that led Wendall to hire me.)

This particular control system managed the weighing and dispensing of the ingredients for making concrete blocks. There was a conical bin suspended over the mixer that was hung on a mechanical scale system. As materials were added to the bin, the needle on a big Cardinal Scale rotated to indicate the weight of the material in the bin.

Diagram of a modified mechanical scale dial
Scale diagram

To automate this process, Mr. Pete came up with the idea of gluing a small magnet (about 1/8″x1/8″x3/4″) to the front of the rotating needle indicator. Then he added several aluminum rod pointers to the face of the scale which could be adjusted to point to specific weight values. Each rod had a magnetic reed switch attached to its back side. As the scale needle rotated, the proximity of the magnet as it passed by a pointer would cause the tiny reed switch to close, completing a circuit. The Scale diagram shows some of the components described here, and their relative locations.

He ran tiny wires from the reed switch pointers to the control panel, and used them to automate the operation. With a single press of a button, the sand conveyor would start adding sand to the bin until the scale’s needle reached the first pointer. The sand conveyor would cut off and the aggregate conveyor would start adding fine rock until the second pointer was reached, at which time the cement conveyor would start adding cement until the third pointer was reached. The bin would then dump its contents into the mixer. Water was added using a water meter, and the rest was more or less manual. The operator would watch it mix until it looked right, then dump the mixture into the block-making machine, and start the weighing process again.

I prepared for writing the program by studying Mr. Pete’s D-sized drawing (about 36″x24″) of the circuit, which consisted mostly of relays and timers. Relays are generally used to take the output of a low-power device, such as the reed switches, and allow it to control a more powerful device, such as a large conveyor motor or a pneumatic valve attached to a cylinder actuator. Timers are used to add preset delays to the process.

A sample diagram of a relay circuit in ladder format
Ladder diagram

Fortunately, the 5TI used relay-based ladder logic as its programming “language”. Ladder logic depicts a circuit as two vertical lines (the plus and minus or hot and neutral voltages) which are connected by horizontal “rungs” which introduce logic in the form of series and parallel contacts (logically equivalent to AND and OR). You “program” the 5TI by entering a list of elements which emulate relays. The 5TI supported inputs, outputs, relays and timers. The Ladder diagram shows an example of a push-button-controlled relay,a timer, and an indicator lamp. (Analysis of this logic is left as an exercise for the reader.)

In my first few weeks at Superior Steel, I had successfully done some troubleshooting of control panel malfunctions using a volt meter and Mr. Pete’s circuit diagram, but when I tried to understand the whole thing, there were a couple of relays whose purpose I didn’t understand. [Narrator: cue the dum-dum-DUM music.] But I had a pretty good idea of how everything was supposed to work, so I moved on.

Since this was the first time I had tried to do a relay-to-5TI conversion, I wanted it to go right. So I built a SIMULATOR! It was a metal chassis with a handfull of toggle switches (to simulate inputs) and neon lamps (to simulate outputs).

Programming console with buttons and alphanumeric indicators
5TI programming console

I connected the 5TI programmer to the 5TI unit and entered the ladder-logic equivalent to the physical relays and timers as I understood them into the 5TI. The photo of the 5TI controller shows what buttons were available, and gives you an idea of how the programming worked. Once I entered the logic steps, I disconnected the programmer and connected the 5TI electrical inputs and outputs to the simulator switches and lights.

Finally, I stepped through the sequence using a switch to start the sequence. I watched the light symbolizing the conveyor for the first material as it lit up, then I used the switch simulating the first scale pointer reed switch to indicate that the first set-point had been reached. The first conveyor light turned off, and the second conveyor light turned on. After a few seconds, I used the switch simulating the second scale pointer reed switch to indicate that the second set-point had been reached. I went through the whole sequence, and it seemed to work as intended.

Satisfied that everything was ready to go, I delivered the 5TI and its input/output chassis to the concrete block plant, and their electrician wired everything in place according to my instructions.

At last the day came to commission the system and mix the first batch.

As the owner and his operator stood watching, I pressed the start button. Lights flashed, a motor starter clonked, the conveyor started up, and the first material began to flow into the bin, just as planned. When the scale needle swung up to the first pointer, the first conveyor stopped, and the second conveyor started, just as expected.

Then something completely unexpected happened. The scale began to fill much more quickly than expected. The needle passed the second pointer, and kept rising. While I was trying to figure out what the heck was going on, the operator was shouting, “Turn it off, turn it off!”

By the time I could get it stopped, sand and rock had overflowed the scale bin, onto the floor, and into the mixer.

We stood there trying to figure out what had gone wrong. The operator pointed out that, just a few seconds after the second conveyor had started, the first conveyor had started back up, dumping excess material in the rapidly filling bin.

My first thought was that the 5TI had run the program wrong. In retrospect, that would have been a pretty ominous explanation. Fortunately that was not what had happened. After a little thought, I realized what the problem was. And what those extra relays were for in Mr. Pete’s original drawing!

I also learned, in memorable fashion, why a simulator is only as good as the accuracy of its representation of reality.

What happened was this: when the second conveyor began to add material to the scale, and the scale needle began to rise, the magnet no longer kept the first reed switch closed. So the program ‘thought’, “I need to add more of the first material.” And so it did.

Mr. Pete’s unexplained relays were latches. Each relay was wired so that, as the measuring cycle went through its steps, when it was activated by the scale pointer, it would hold itself on until the end of the process. It served as an electrical memory that indicated, “We don’t need any more of this material on this cycle, even if the scale pointer switch opens back up.” (The first line on the ladder diagram is a similar latching circuit for the “Start” pushbutton.)

So why didn’t I catch this logical fallacy on the simulator? Because the switches I used were toggle switches, not momentary switches. To simulate the actual operation of the scale circuit, I would have had to have turned off the first switch to emulate the action of the reed switch as the scale needle increased. If I had thought of it.

Back at the concrete block plant, it fortunately only took a few minutes to add latching relays to the 5TI program.

It took somewhat longer to convince the owner and the operator that it wouldn’t happen again next time.

And it took even longer to shovel up the excess rock and sand from in and around the mixer. They didn’t ask me to help, but since it was fundamentally my fault, I grabbed a shovel and did my part.

I can’t say that I never mis-used a simulator after that, but I will say I was considerably more cautious.


If you are interested in more information on the 5TI or ladder logic, here are a few references:

A 5TI article on a cool electronics archive site (and the source of the 5TI programmer photo):

http://www.decadecounter.com/vta/articleview.php?item=172

A description of ladder logic diagrams:

https://en.wikipedia.org/wiki/Ladder_logic

And, as usual, Wikipedia has a very comprehensive article on Simulation: https://en.wikipedia.org/wiki/Simulation

The father of Information Theory, Claude Shannon, gets the credit for showing how series and parallel relay contacts are equivalent to AND and OR logic operators. https://en.wikipedia.org/wiki/Claude_Shannon