Concatenating WiShield packets: string size limitation?

[Uberhund]

My WiShield Web client application must receive and process responses from its server that vary in length from a minimum of 1024 to over 4028 bytes. Because these are XML formatted responses, the entire response must be processed as one message because the WiShield return function is not re-entrant. In other words, the return function cannot easily know where it left off from the previous, truncated batch of XML commands.

Therefore, it is necessary to concatenate all responses from the WiShield into a single string for processing once all packets have been returned.

The Arduino '328 sketch appears to properly concatenate responses when the total number of bytes is 500 or fewer, however, the sketch hangs the Arduino when I set the size of the buffer to anything greater than 500 bytes.

Any suggestions what I might be doing wrong, or how I might extend the buffer size to 4028 or more?

Here's the sketch, adapted from the WiShield Weather GET sketch for fetching weather reports from the LAX server.

Code: Select all


#include <WiServer.h>
#include <NewSoftSerial.h>

#define WIRELESS_MODE_INFRA   1
#define WIRELESS_MODE_ADHOC   2

// Wireless configuration parameters ----------------------------------------
unsigned char local_ip[] = {192,168,1,10};   // IP address of WiShield
unsigned char gateway_ip[] = {192,168,1,1};   // router or gateway IP address
unsigned char subnet_mask[] = {255,255,255,0};   // subnet mask for the local network
const prog_char ssid[] PROGMEM = {"SSIDHERE"};      // max 32 bytes
unsigned char security_type = 3;   // 0 - open; 1 - WEP; 2 - WPA; 3 - WPA2

// WPA/WPA2 passphrase
const prog_char security_passphrase[] PROGMEM = {"PASSPHRASEHERE"};   // max 64 characters

// WEP 128-bit keys
// sample HEX keys
prog_uchar wep_keys[1];

// setup the wireless mode
// infrastructure - connect to AP
// adhoc - connect to another WiFi device
unsigned char wireless_mode = WIRELESS_MODE_INFRA;

unsigned char ssid_len;
unsigned char security_passphrase_len;
// End of wireless configuration parameters ----------------------------------------

#define bufferLimit 500                             //At 500, everything works great; at >500, '328 hangs
char sQueueBuffer[bufferLimit];
int iQueuePtr=0;

// Function that prints and processes data packets from the server
void printData(char* data, int len) {

  if(len==0){                                            //All packets have been returned; let's wrap this up
    Serial.print("Total Queue size: "); Serial.println(iQueuePtr);
    sQueueBuffer[iQueuePtr]=NULL;
    Serial.print("Length of buffer: ");Serial.println(strlen(sQueueBuffer));
    Serial.print(sQueueBuffer);
    Serial.println();
    Serial.println("End of XML Queue");
    iQueuePtr=0;
    return;
  }

  // Print the data returned by the server
  // Note that the data is not null-terminated, may be broken up into smaller packets, and
  // includes the HTTP header.
  Serial.print("Packet len is: "); Serial.println(len);

  while (len-- > 0) {
    if(iQueuePtr<bufferLimit)               //limit # of  transfers so buffer doesn't overrun
      sQueueBuffer[iQueuePtr++]=*(data++);  //transfer data to accumulation buffer
  }

}


// IP Address for www.weather.gov 
//uint8 ip[] = {140,90,113,200};
// IP for NIST
//uint8 ip[]={64,90,182,55}; //nist1-ny.ustiming.org
// IP for MySERVER
uint8 ip[]={XX,XX,XX,XX};

// A request that gets the latest METAR weather data for LAX
//GETrequest getWeather(ip, 80, "www.weather.gov", "/data/METAR/KLAX.1.txt");
// request for time from NIST
//GETrequest getWeather(ip, 13, "nist1-ny.ustiming.org", "/");
//request from MyTango
GETrequest getWeather(ip,80,"www.server.com","SSI INFO HERE");


void setup() {
 
  Serial.begin(9600);
  Serial.println("Initializing...");

    // Initialize WiServer (we'll pass NULL for the page serving function since we don't need to serve web pages)
  WiServer.init(NULL);
 
  // Enable Serial output and ask WiServer to generate log messages (optional)
  WiServer.enableVerboseMode(true);
  // Have the processData function called when data is returned by the server
  getWeather.setReturnFunc(printData);
  Serial.println("Setup complete"); 
    Serial.println("Submitting GET");
    getWeather.submit();     
}


void loop(){
  WiServer.server_task();
  if(Serial.available()){
    while (Serial.available())
      char cFoo=Serial.read();
    Serial.println("re-submitting...");
    getWeather.submit();
  }
  delay(10);
}



Thanks for your help

ub3rhund

[GregEigsti]

As I see it there are a couple of choices - you can either reduce the code/data size that your script is using to allow you to create your large buffer (though I don't know if you will be able to reduce the size sufficiently for such a large scratch buffer) or you can parse the data on the fly and not use the large buffer. Nothing that cannot be done with a little pointer management and a dash of state machine

Greg

[Uberhund]

Thanks for your input, Greg. I value your opinion.

Unfortunately, though, the state-machine/pointer approach greatly expands the code, uses more of the program space, and, without breakpoint-singlestep emulation (an Arduino shortcoming), is *extremely* hard to debug.

Consequently, I had to give up on the wireless approach and go with a wired Ethernet shield product. Doing so simplifies the state-machine (xPort, for one, returns entire lines rather than truncated packets; XML tends to be organized by lines), plays nice with a larger set of routers than the WiShield, AND I get user-friendly DHCP as well.

I look forward to future versions of WiShield, perhaps for my next project. Your support has been terrific.

[GregEigsti]

BTW - DHCP and DNS (as well as other goodies) work in the AsyncLabs user contrib branch And there is sample code to show how to get started.
Also, you can read incoming data line by line when using the uIP APIs; WiServer abstracts those away - flexibility is lost with the additional simplicity.

So you have not convinced me - but its your project so that does not really matter - hahahaha

Have fun! Thats what it is all about! And I hope your project turns out well!
Greg

[Uberhund]

Very good to know, Greg. Meanwhile, I'll take a look at the APIs. That could be a big break through for the wireless version. Thanks for the tip!

[GregEigsti]

The SocketApp example contains a call to PSOCK_READTO(&s->p, '\n') in the handle_connection() function in socketapp.c - passing '\n' as the second parameter gets you line by line buffers o' data. My signature contains links to the user contrib branch (DNS/DHCP/etc.) and the uIP docs (the uIP bible).

Greg