Remote pump control ideas.

[stj]

Re: Remote pump control ideas.

it would help if you call them master and slaves.
master commands, slaves respond when spoken to.

[Dannyx]

Re: Remote pump control ideas.

it would help if you call them master and slaves.
master commands, slaves respond when spoken to.

Makes sense, but can there be multiple masters ? I'm sure of it...

[stj]

Re: Remote pump control ideas.

yes, if they dont clash on the network.
so dont poll the slaves multiple times a second and it should work fine

[Curious.George]

Re: Remote pump control ideas.

Sort-of....This was the idea that came to my mind after what STJ suggested (EDIT: OK, no, he didn't actually suggest THIS, but I worked around it): have multiple servers which listen for the single incoming client - sounds stupid, but strictly from the perspective of the hardware I'm working with at the moment, it's the only workaround that I, the average Joe, sees Hence why I love discussing stuff like this and learning how more advanced chaps do it.

A "server" is typically a single device. It can handle multiple clients, concurrently, using a number of different implementation mechanisms.

You would, thus, have a single node that exports a particular "service". Clients on each of the other nodes would then connect to that service.

Note that -- with ethernet protocols -- you always want to be a good "net citizen" as it's almost certain that, sooner or later, you will have to coexist with other protocols/services running on that network. So, you want to "do things right" instead of ad hoc.

[Dannyx]

Re: Remote pump control ideas.

A "server" is typically a single device. It can handle multiple clients, concurrently, using a number of different implementation mechanisms.

So I thought. Makes perfect sense and is completely logical, but unfortunately this is slightly limited in the case of the Wiz W5x00 series the arduino library works with, that's why I thought I'd look into more advanced ways of doing, but not too over the top - I'm not planning to become a master coder overnight. Just enough to save my life and design simple projects like these to keep my boss happy while at the same time learning from mistakes

I had a look at some tutorial vids on the STM32 series....it's actually overwhelming. The difficulty curve is very steep. So they use their own IDE - "Eclipse" (much like the Arduino has its own), but since this is not aimed at beginners, what I noticed is that it likes to dump everything it uses in the code window, which the Arduino IDE doesn't do, since it spares the headaches of newbies. For instance, it includes the stuff which is normally hidden behind the libraries, like the parameters for UART and other peripherals - it's all there in the code window, so you have to really know what you're doing, what to modify and what not to modify. I was able to follow along and understand the "while(1)" loop is the equivalent of void loop() in Arduino, where I'd put my main code. Not sure if it'd be as easy as copying and pasting though - the pin numbering wouldn't match for one thing and even after that, I'm not sure all the functions are compatible...

The part I'm mostly interested in, despite being too early, is how you deploy ethernet on one of those. Even if I include the STM32 boards in the Arduino IDE (which CAN be done apparently, for beginners who wish to play with more powerful boards while still maintaining the simple IDE), the ethernet library for the Arduino is not going to work here because it's not using the same chipset, so I have to use the Eclipse for this.....sucks that time is so scarce and everybody wants stuff done yesterday, but I guess that's the price I pay for embarking on such a journey

[Curious.George]

Re: Remote pump control ideas.

So I thought. Makes perfect sense and is completely logical, but unfortunately this is slightly limited in the case of the Wiz W5x00 series the arduino library works with, that's why I thought I'd look into more advanced ways of doing, but not too over the top - I'm not planning to become a master coder overnight. Just enough to save my life and design simple projects like these to keep my boss happy while at the same time learning from mistakes

I have no idea what their library supports. My (uninformed) opinion of these devices is that they aren't really intended for "serious" work.

I had a look at some tutorial vids on the STM32 series....it's actually overwhelming. The difficulty curve is very steep. So they use their own IDE - "Eclipse" (much like the Arduino has its own), but since this is not aimed at beginners, what I noticed is that it likes to dump everything it uses in the code window, which the Arduino IDE doesn't do, since it spares the headaches of newbies. For instance, it includes the stuff which is normally hidden behind the libraries, like the parameters for UART and other peripherals - it's all there in the code window, so you have to really know what you're doing, what to modify and what not to modify. I was able to follow along and understand the "while(1)" loop is the equivalent of void loop() in Arduino, where I'd put my main code. Not sure if it'd be as easy as copying and pasting though - the pin numbering wouldn't match for one thing and even after that, I'm not sure all the functions are compatible...

The part I'm mostly interested in, despite being too early, is how you deploy ethernet on one of those. Even if I include the STM32 boards in the Arduino IDE (which CAN be done apparently, for beginners who wish to play with more powerful boards while still maintaining the simple IDE), the ethernet library for the Arduino is not going to work here because it's not using the same chipset, so I have to use the Eclipse for this.....sucks that time is so scarce and everybody wants stuff done yesterday, but I guess that's the price I pay for embarking on such a journey

"Eclipse" is pretty much the de facto IDE for most "serious" software development work.

If you plan on using ethernet to talk between devices/nodes, you have two ways to go.

If you want to handle a lot of the details of the communication protocol yourself, you don't need to rely on having a full network stack. You can, instead, build a protocol on UDP which can be implemented "cheaply".

But, UDP leaves lots of details up to you to implement (message sequencing, retries, connection handling, etc.). It's essentially just pushing bytes down the wire and hoping that your devices manage to capture them as they pass. If a message gets corrupted (or missed!), then YOU have to implement some protocol for resending it.

[This might not be an issue if you have lightly loaded devices on the same network segment with little other network traffic; but, stick a router in there or other protocol traffic on the wire and its harder to guarantee correct operation -- that's why TCP was invented!]

If you want a "virtual connection" between a program on one device (a client) and another program on another device (a server), then TCP makes your life REALLY easy; you talk directly to the other party and the network stack handles all the details of error correction, timeouts, packet reassembly, retransmissions, routing, etc.

A (developer) friendlier approach might be to use something like Inferno. But, you'd need to find a port for your chipset -- likely not available for "oddball" devices. (I think an rPi port may be available)

[Dannyx]

Re: Remote pump control ideas.

I have no idea what their library supports. My (uninformed) opinion of these devices is that they aren't really intended for "serious" work.

Well, no, they're not - the Arduinos are for people getting started with this sort of thing. You CAN get into more complex stuff with them, but they're not industrial-grade. They're useful for small projects like these which do very little.

[Dannyx]

Re: Remote pump control ideas.

BTW, any suggestions for small form-factor boards with wired Ethernet ? The Nucleo 144 is WAY too big physically for building small "clients" - it's a waste of both space and hardware to just read 1-2 pins on it. I imagine something like an ESP8266 but with wired Ethernet...or a more hacky way: an arduino Pro Mini or Nano with an ethernet shield patched onto it with jumper leads, which is what I currently have on my workbench to test all these projects

[stj]

Re: Remote pump control ideas.

ARDUINO is not hardware, it's the software platform which is based on the GCC C compiler with lots of libraries to make it simple to manipulate the hardware.

[Curious.George]

Re: Remote pump control ideas.

BTW, any suggestions for small form-factor boards with wired Ethernet ? The Nucleo 144 is WAY too big physically for building small "clients" - it's a waste of both space and hardware to just read 1-2 pins on it. I imagine something like an ESP8266 but with wired Ethernet...or a more hacky way: an arduino Pro Mini or Nano with an ethernet shield patched onto it with jumper leads, which is what I currently have on my workbench to test all these projects

(sigh)

Sorry, I've lost track of what the desired goal of all of this is (and I'm not keen on reading 80 posts to sort it out!).

Think about what you are trying to do "in the abstract".

Think about the constraints (physical, financial, temporal) that you have to address.

Think about the resources (money, time, expertise) that you have available.

Remember, wired ethernet will require a switch (or switches), somewhere. Accommodating that switch can suggest other communication topologies (e.g., it's a physical star but a logical bus; perhaps consider a physical star AND logical star? i.e., have everything plug into a "master")

Also remember that you can implement networks with a variety of hardware technologies and protocols. E.g., you could implement an RS485 network and layer a token-passing protocol atop it (to eliminate the concept of a "master"). Or, a TDM access scheme driven by the unique ID of each node. Or...

You may find that your data rates are low enough that slower, less complex hardware can address your needs. Or, may find the wire distances involved make some solutions "challenging".

The appeal that "ethernet" has is that it is a standard base for many protocols and services "out-of-the-box". E.g., you can layer a web service atop a TCP/IP stack in a jiffy. Trying to offer the same service on some other protocol/hardware means a fair bit of work (esp if you want to use standard browers)

You may not need that flexibility. But, be sure that you WON'T at some future date! It's really hard to tell an employer/client that you've got to scrap their existing hardware AND software "just" to add one of these sorts of capabilities.

[Dannyx]

Re: Remote pump control ideas.

Sorry, I've lost track of what the desired goal of all of this is (and I'm not keen on reading 80 posts to sort it out!).

Yeah, it's my fault. I said I wouldn't shoehorn another project in here which has nothing to do with the original one, so let's fold back.

It all started with trying to control a three-phase pump based on inputs from a mechanical switch which is located kms away from the pump. What I have to work with is a GPON and internet connection in each shack, from a hardware perspective...I have no idea what the boys over at our ISP department actually DO with these, but it's their job to make the two boards "see" each other, while I on the other hand, have to make them "talk" TO each other....which I did successfully on the bench (even with some rudimentary checks to make sure each run/stop command was actually received, as previously discussed) though no programmer would hurry to praise me for the frankenstein I created

[Curious.George]

Re: Remote pump control ideas.

Yeah, it's my fault. I said I wouldn't shoehorn another project in here which has nothing to do with the original one, so let's fold back.

It all started with trying to control a three-phase pump based on inputs from a mechanical switch which is located kms away from the pump. What I have to work with is a GPON and internet connection in each shack, from a hardware perspective...

Presumably, the fiber is terminated into something "electronic" that presents a standard ethernet interface? Or, do you have to design hardware to support the optical transducers (presumably via SFP)?

Assuming you have some means of connecting an traditional "wired" interface...

I have no idea what the boys over at our ISP department actually DO with these, but it's their job to make the two boards "see" each other, while I on the other hand, have to make them "talk" TO each other....which I did successfully on the bench (even with some rudimentary checks to make sure each run/stop command was actually received, as previously discussed) though no programmer would hurry to praise me for the frankenstein I created

As you can't vouch for the integrity of the link at all times, the first thing you need to do is decide what the fail safe (or secure) condition will be at the far (remote) end. "Something" in your code will have to drive the pump controls to that condition when somethingelse has determined that the virtual link to the mechanical switch is no longer to be trusted.

[don't forget to figure in the possibility of active threats]

If this is a dedicated (optical) link, then you don't have to worry about other traffic on the "wire" unless YOU put it there! Nor any routing.

And, if it is dedicated, you can likely assume any datagram has an equal chance of transiting the link.

So, design some magic packets to encode your desired actions (state of switch, state of pump) and pass them back and forth over UDP. You can include an explicit packet number to reassure yourself that "this" is the correct "next packet" -- to guard against dropped/corrupted packets.

You can take advantage of the entire datagrams' payload to encode the information to ensure corruption can be detected. I.e., if you just encode the command in one bit -- on/off -- then that one bit being corrupted leaves you in an compromised state. By contrast, if you encode the one bit in a field of dozens (or hundreds) you can deduce its intended value even if a large number of bits are corrupted (google "Hamming Distance").

Or, you could opt to NOT act (or, drive the system to its "safe" state) if you don't receive an "ideal" datagram.

[Dannyx]

Re: Remote pump control ideas.

Or, do you have to design hardware to support the optical transducers (presumably via SFP)?

No - the network side is already in place. Fibers terminate somewhere in our datacenter, or whatever.... I was hoping it'd be a simple Media Converter box at each end, with the fiber link simply running between them (so a layer 1 setup, if I'm not mistaken - no "brains") where I could simply assign static addresses to both the client and server and that would be it - case closed - but instead, some routing has to be done for them to see each other, since the fibers each run somewhere in our head office where God know what happens next - probably goes to some "OLT" (if I'm calling it right)...

As you can't vouch for the integrity of the link at all times, the first thing you need to do is decide what the fail safe (or secure) condition will be at the far (remote) end. "Something" in your code will have to drive the pump controls to that condition when somethingelse has determined that the virtual link to the mechanical switch is no longer to be trusted.

Yes, I did this, though in a very crude way, like I said: I set up the server to send out "ping" messages to the client every X seconds. If there's no reply after Y retries, it means something went wrong (fiber broke...etc) and the server should stop the pump to keep the tank from overflowing (since it won't know at this moment if the float switch has done anything). Same goes for the run/stop commands - after sending one, the client waits for an "acknowledge" message back from the server. If it doesn't get one, it send out the run/stop command again and again until it does. I worked this out on my own because I noticed sometimes the commands get corrupted and don't come in "loud and clear" - parts of it go missing and the code doesn't know what to do with it...Please see post #23 where I talk about what happens. It's all based on an example sketch I found somewhere, which I then adapted, hence why I said it's not too advanced...

[Curious.George]

Re: Remote pump control ideas.

No - the network side is already in place. Fibers terminate somewhere in our datacenter, or whatever.... I was hoping it'd be a simple Media Converter box at each end, with the fiber link simply running between them (so a layer 1 setup, if I'm not mistaken - no "brains") where I could simply assign static addresses to both the client and server and that would be it - case closed - but instead, some routing has to be done for them to see each other, since the fibers each run somewhere in our head office where God know what happens next - probably goes to some "OLT" (if I'm calling it right)...

My point has to do with whether the end of the fiber is "the other end of the link" (dedicated between "switches" and "pump") or, just a "network drop"?

In the former case, you "own the wire" and can do whatever you want without worrying about interference. In the latter, you can't directly control what is pushed down the "wire".

Yes, I did this, though in a very crude way, like I said: I set up the server to send out "ping" messages to the client every X seconds. If there's no reply after Y retries, it means something went wrong (fiber broke...etc) and the server should stop the pump to keep the tank from overflowing (since it won't know at this moment if the float switch has done anything). Same goes for the run/stop commands - after sending one, the client waits for an "acknowledge" message back from the server. If it doesn't get one, it send out the run/stop command again and again until it does. I worked this out on my own because I noticed sometimes the commands get corrupted and don't come in "loud and clear" - parts of it go missing and the code doesn't know what to do with it...Please see post #23 where I talk about what happens. It's all based on an example sketch I found somewhere, which I then adapted, hence why I said it's not too advanced...

You want to ensure the "wire" is correctly carrying the "state" of the system: switch=on, pump on vs. switch off, pump off. The other states are normally transitional (switch on, pump off -- i.e., the switch was JUST turned on and the pump hasn't yet received the command to turn on; switch off, pump on -- switch was just turned off and pump hasn't yet been told to turn off)

However, there are other cases where these "illegal" states can persist. For example, if the commands don't get through intact (because the link is flakey) or if the pump decides to "fault safe" (despite being told to stay on).

So, there's a third party involved that you need to validate -- the link itself.

To do this, you want to pass messages that each side should EXPECT to encounter and let them verify that all is as they EXPECT it to be.

["turn pump on" and "turn pump off" might be LEGAL messages but neither side KNOWS when to expect either of these!]

If, OTOH, you regularly exchange messages (or, portions of messages) that contain EXPECTED content, it lets the two ends convince themselves that the link is functioning properly.

Note that you can do this with content and timing. I.e., if you expect to receive a message every 10 seconds, then failing to receive one tells you that the link is faulty or the other end is having troubles. (OTOH, if you had no expectation of receiving a message until the switch state had changed, you would never know if the state had changed and you'd not been informed due to some problem!)

While a message may contain "unexpected" content (i.e., you don't know if it's going to say "on" or "off"), you can still reassure yourself by verifying other portions of the message contain the correct EXPECTED content.

Commonly, you add sequence numbers that let each end reassure themselves that they have seen all of the traffic that was destined for them.

"This is message #145. Pump on"
"This is message #146. Pump on"
"This is message #147. Pump on"
"This is message #148. Pump off"

So, if you last saw #146 and now see #142, you can assume something is wonky. (maybe the 2 should have been a 7? maybe the 6 really was a 1?)

The whole point is to reassure yourself that the link appears to be functioning correctly.

Note that you should also add common sense to your code. I.e., if you are told to turn the pump on, then off, then on, then off just a few seconds apart, something is likely not right. Likewise, if the pump is told to turn on and never told to turn off 300 hours later...

[redwire]

Re: Remote pump control ideas.

Don't forget the tank side does not have unlimited power.
It has to wake up a do a session when the tank empty switch is asserted, or every say 1/2 hour as well, get on the network and try send a message to the pump station. (I'm confused if the network is GSM cellphone or Wifi or Ethernet or RS-485 etc).

After the tank is full, that message goes out to cause the pump controller to shut off the pump. If ACK then we're done and go back to sleep.
When the pump is running there needs to be a com link verification, like a ping and response or the pump shuts off by itself if the link goes down for XX seconds.

DannyX you have to try not to do too much here. Even moving to a new MCU and hardware platform is a huge leap. If you are pushed for time then the Arduino platform works well enough. Learning STM32 and the IDE is a lot of work unless you have the time.

I use MODBUS for the simplest network, ASCII if you really want to be able to snoop on the message traffic for debugging. The CRC checksum helps with scrambled messages. I add a back-door so I can send my own remote queries/commands such as battery voltage, signal strength RSSI (for antenna alignment), number of com fails etc. without driving to the site.

[Dannyx]

Re: Remote pump control ideas.

I'm confused if the network is GSM cellphone or Wifi or Ethernet or RS-485 etc

It's Ethernet over fiber - the same kind we have in our homes. I don't know the correct terminology for this stuff In complete layman terms, it gives each shack access to the internet through those ZTE GPONs the guys installed, so I think it's a "network drop", the way Curious.George refers to it.

When the pump is running there needs to be a com link verification, like a ping and response or the pump shuts off by itself if the link goes down for XX seconds.

Yes, it does this too, exactly as you describe, even when the pump's not on: each failed ping adds +1 to a counter. When it reaches 10 failures (or however many I feel like it's a sensible number), the pump turns off and also sends an SMS message to whoever's in charge, since I'll be keeping that GSM alarm module at the pump end and use it for this purpose (the one this whole setup originally ran on). I pull one of its input pins LOW, which the micro can successfully do, since I'll be powering them both off the same supply and it sends out an SMS message to a specified number. I'll also use it to monitor the mains, since I'll be installing a MeanWell ADD-155A supply which has a battery backup, so communications can stay on even with no mains, enough to alert someone.

Now: Modbus has always been confusing AF to me, now that you brought it up, because I can't work out how to implement it: I understand the principle, but how does it work physically ? I'm talking about the hardware. Can I use it over ethernet just like that ? Do you have to run a wire between the master and slave ? Take my scenario for instance...how would any of you go about this project ?

[Dannyx]

Re: Remote pump control ideas.

I decided to have a look at Modbus as well, since like I said - it's always been a confusing topic to me. I woke up feeling motivated today and managed to understand it a bit better I downloaded an example for the Arduino library to mess with. It seems to do what I need it to do: it even checks if the commands were accepted and reports back if there were errors. When using it over TCP (as opposted to RS485), it seems to operate "backwards": a client commands a server, though this is only mentioned very briefly HERE, so I'm not sure if I understood it wrong. Either way, I'll play with it for the OTHER project, which I won't mention here again

I hit a snag right away though: I had no idea how much memory the Modbus library would eat up, so I ran out of space on the Pro Mini "Global variables use 2207 bytes (107%) of dynamic memory, leaving -159 bytes for local variables. Maximum is 2048 bytes"

[stj]

Re: Remote pump control ideas.

get yourself a blackpill

[Dannyx]

Re: Remote pump control ideas.

You mean THIS ?
At first I thought it was some sexual innuendo or something, since that's what I got among the first results on dear Google It wasn't until I punched in "STM" in the search that the board came up

[Curious.George]

Re: Remote pump control ideas.

When using it over TCP (as opposted to RS485), it seems to operate "backwards": a client commands a server, though this is only mentioned very briefly

That's how a client-server system operates. The server exports a service. Clients make demands on that service.

DNS is a service. A client sends a request to the DNS server: "Give me the IP address associated with Google.com". The server responds with the appropriate "answer".

HTTPd is a service. A client (someone using a web browser) issues a request to a particular HTTPd server: "Give me your top level index.html page". The web server responds with a copy of that page.

I hit a snag right away though: I had no idea how much memory the Modbus library would eat up, so I ran out of space on the Pro Mini "Global variables use 2207 bytes (107%) of dynamic memory, leaving -159 bytes for local variables. Maximum is 2048 bytes"

The reason many applications dumb down to run on UDP -- instead of TCP -- is the cost of the network stack. (UDP can be implemented in hundreds of bytes vs. kilobytes for TCP -- plus timers, etc.)