Restored Connection With shard7

[shard7]

Hi Everyone,

Sorry for being off the grid for a while, having a young baby and lots to do at work and home will do that.

First, I'm thrilled that so many people are finding great uses for WiServer. It started as just a bit of code that I cobbled to together for my own use, but looking at this forum it's clearly become much more than that. Also, a big THANK YOU to several of you who've been helping folks out on this forum and answering their questions.

I agree that the lost connections are a significant issue with the WiShield, and the current WiServer code just blindly assumes that the connection is always there which is pretty lame. I've noticed that several ideas have been tried to address this shortcoming, and I'd like to compile a summary of those to use as the starting point for a permanent fix. So, if you've created a change to the WiServer code in an effort to improve connection stability, please post your findings on this thread. Please include any relevant code changes, a description of how your fix is intended to work, and how well it seems to perform.

Thanks in advance for your help!

shard7

[GregEigsti]

having a young baby and lots to do at work and home will do that.

Congrats! I have a 5 yo and a 9 yo that I could send to you for a few months so that you can get a feel for what you are in store for!

Here is a good thread to look at; I tried to (hackishly) address a disconnection issue with WiServer; while it worked fine against my repro case others (aerodyno?) reported that it did not fix everything for him (if I remember correctly). The only reason that I point this all out to you is that it did uncover an issue where WiServer's POST/GET/etc. classes make use of an "active" variable that was not getting reset during a "hail mary" board reset and then causing problems down the line (they thought they were still "active" after a reset and everything was just really confused). Anyway seems to me like this might be helpful data in your quest

Greg

[John_Ryan]

Correction! The culprit is malloc and/or free .. as you were.

[shard7]

Hi Everyone,

Thanks for all the info about the connection issues, I think I may have a solution.

Somewhat like hruska's changes, I've added a simple state machine to the WiServer to monitor the connection and restart things as needed. What's different is that it's all handled through the server_task function so your sketch loop stays nice and simple.

The first change is that the WiShield is no longer started by the WiServer.init() call, and it will promptly return (it's asynchronous) and allow the rest of your code to run. Instead, initializing the WiShield and establishing the connection is performed by the server_task function, and thus it can automatically do so again if needed in the event of a lost connection.

When a connection is lost, uip_abort is called and the uip stack is run so that any pending requests get cleaned up. This means that the 'active' flag on a POST or GET request should get reset correctly. The state machine is then reset to the initial disconnected state in order to restart the connection.

I did notice that the WiShield likes to establish a connection within a few seconds of being initialized. If a minute or so passes before the wireless network is active, it doesn't seem to connect. To address that behavior, the state machine waits up to 15 seconds for a connection after initializing the WiShield; if no connection is made, it re-initializes the WiShield and starts over.

Calls to server_task are still asynchronous, so it returns promptly even if WiServer is waiting for a connection. However, there are instances where you may want to wait for connection before doing other things. Therefore, server_task now returns a boolean value to let you know if there's a valid connection, and there's also a new function called checkConnection(int seconds) that allows you to query the connection and if necessary, wait for a certain period of time for a connection. You can call this function with 0 to just check the state and return, or you can use -1 to wait indefinitely until there's a connection.

This code is still experimental and has debug printlns all over the place, but I'm posting it here in the hopes that a few brave soles will try it out and let me know how well it works. Only WiServer.h and WiServer.cpp are affected. Once it's been proven to address the issues, I'll clean it up and post to the official source repo on Github.

Enjoy!

Mark

[John_Ryan]

Well done!

I'll take it for a spin early next week, although I haven't had connection issues in WiServer mode with Greg's fix plus increasing the number of connections. I also use an "any-URL" fall-back which seems to eliminate the problem of good and evil bots visiting and swamping the controller with requests.

And now while I have your attention, I also need testers for WiPiler!! so if you get a chance head on over here:-

http://www.wipiler.com/

It's an web-based command line editor with a small set of commands. If testing doesn't uncover too many issues then I'll be adding a command set for data logging. The LOG command will read data from a pin at x intervals, and write it with the current date and time to the WiPiler users xml file. I noticed a lot of work had been going on over at Arduino to do the same thing using SD cards, so I think a web based version that doesn't need extra hardware or massive libraries should be well received and hopefully it'll give people another reason to buy WiFi = )

[GregEigsti]

Nice work all!

Greg

[hruska]

Great to see this all coming full circle and getting into the actual release!

[John_Ryan]

I've had my Arduino/WiShield constantly connected now for 4330 minutes (72 hours), which isn't unusual. What is unusual, is a Wipiler RAM query returned -26279 bytes

Prior to that, and after staying connected for "up to" 1365 minutes (22 hours), WiPiler was reporting available RAM of 394 bytes, which is the same amount it reports when the controller is powered up. Considering the sketch, 394 bytes I would think is quite realistic for an Atmega 328.

So.. something weird is going on inside the Arduino the longer the device stays powered. Below is the function that returns the amount of available RAM - can anyone see anything that might cause RAM to mis-report the longer the device stays connected?

I've posted this here because if the RAM function is rendered useless over time, it could be an indication that a problem exists when an Arduino stays connected to a WiShield for lengthly periods, and that such a problem could go on to manifest as dropped connections, or other unexplained glitches.

Code: Select all

// Variables
extern int  __bss_end; 
extern int  *__brkval;

// FUNCTION: return the number of bytes currently free in RAM     
int freemem(){
    int free_memory;
    if((int)__brkval == 0)
        free_memory = ((int)&free_memory) - ((int)&__bss_end);
    else
        free_memory = ((int)&free_memory) - ((int)__brkval);
    return free_memory;
}


[GregEigsti]

John_Ryan, I'm sure you have seen this thread - wink, wink. But there seem to be some additional memory reporting routines here that may be of interest. Personally I stay the hell away from the heap on the 328 - chant after me - its just not worth it, its just not worth it

Take a look at the memb.c/h source files in the WiShield distro; does not look like they are used anywhere in the WiShield code but they are interesting nonetheless - a memory manager for statically allocated memory blocks. And if memory serves me right everything in the WiShield (sans WiServer) code is statically allocated. (ok, I will not try to get detailed here because I am not very familiar with the 328 memory arch).

So... I wonder if WiServer was not using the heap if things may be a bit more "stable"? WiServer uses malloc() in two places - I wonder if it could be temporarily be neutered to not use the heap and tested with your memory hoo-haw? It could be that memory is being leaked in the one of two places that WiServer mallocs()? Funny thing is that base64encode() returns a malloc()'ed buffer - wonder how many casual users of base64encode() are actually ever bothering to free that memory?

Greg

[GregEigsti]

John_Ryan
Been perusing the WiServer source and as mentioned there are two places where malloc occurs - in base64encode and in response to a new connection to a WiServer server app (in processLine). Assuming you are using WiServer in the server mode? Wondering if this second bit of malloc'ed data is not being freed in some circumstances? Would be pretty trivial to instrument the malloc and free with Serial.println()'s. Probably a good idea to instrument all places where that particular pointer gets set to NULL as well. Then comparing the malloc/frees/"set to NULLs" it would quickly become apparent if there was a leak.

Dunno - its easier for me to type this to you than to actually get up, walk downstairs and grab a spare WiShield rig. I'll start concentrating and see if I can levitate one up to me

Edit - so I got up off my lazy butt and gave it a try; instrumented WiServer.cpp and also added a call to John_Ryan's memory function. Tried a bunch of different URLs (valid, long, malformed, etc.) to the example simple server app running on the WShield and found nothing that looked like unmatched malloc and frees. Additionally the memory function returned 602 every time. Of course this was over the course of 10 minutes and I did not try hitting the WiShield simultaneously from multiple machines (or processes). Also should add that this was all with the released version of WiServer and not shard7's latest updates

And the real reason I did not get up earlier to play with this was because my sweet little daughter was curled up next to me sleeping and I did not want to wake her. Yup, the softer side of Greg

Greg