WiServer serving larger files

Discussions related to the WiServer add-on code for WiShield. WiServer is a friendly, easy-to-use front end for webserver and webclient usages.

Moderator: shard7

WiServer serving larger files

Postby Dino » Fri Sep 17, 2010 8:16 pm

Hey folks! I've been working on a wireless datalogger using the WiShield 2.0 and a microSD reader with the SS attached to Pin 7.

I have the server up and running on my home network and it will list any files in the root directory of the SD card as links and when you click on the links the browser will spit out whatever is in the file.

It's working GREAT with smaller text files (csv). I've been testing with larger file sizes and it's exponentially getting slower as the file size goes up. Right now a 30kb text file takes about 30-40 sec to open or download.

Is there any way to up my RAM (its using a lot) or modify how much data is being sent per "send" on the WiShield?

ALSO, any pointers on optimizing the code would be nice. This is my first Arduino project. Thanks for any help!

Code: Select all
/*
* This sketch will list all files in the root directory and
* then do a recursive list of all directories on the SD card.
*
*/

#include <SdFat.h>
#include <SdFatUtil.h>
#include <WiServer.h>
#include <string.h>

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

// store error strings in flash to save RAM
#define error(s) error_P(PSTR(s))
#define WIRELESS_MODE_INFRA   1
#define WIRELESS_MODE_ADHOC   2

// Wireless configuration parameters ----------------------------------------
unsigned char local_ip[] = {
  192,168,1,2};   // IP address of WiShield
unsigned char gateway_ip[] = {
  192,168,1,1};   // router or gateway IP address
unsigned char subnet_mask[] = {
  255,255,255,0};   // subnet mask for the local network
const prog_char ssid[] PROGMEM = {
  "linksys"};      // max 32 bytes

unsigned char security_type = 0;   // 0 - open; 1 - WEP; 2 - WPA; 3 - WPA2

// WPA/WPA2 passphrase
const prog_char security_passphrase[] PROGMEM = {
  "12345678"};   // max 64 characters

// WEP 128-bit keys
// sample HEX keys
prog_uchar wep_keys[] PROGMEM = {
  0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d,   // Key 0
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   // Key 1
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   // Key 2
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00   // Key 3
};

// setup the wireless mode
// infrastructure - connect to AP
// adhoc - connect to another WiFi device
unsigned char wireless_mode = WIRELESS_MODE_INFRA;

unsigned char ssid_len;
unsigned char security_passphrase_len;
// End of wireless configuration parameters --------------------------------------------------------------------------------------

void error_P(const char* str) {
  PgmPrint("error: ");
  SerialPrintln_P(str);
  if (card.errorCode()) {
    PgmPrint("SD error: ");
    Serial.print(card.errorCode(), HEX);
    Serial.print(',');
    Serial.println(card.errorData(), HEX);
  }
  while(1);
}

// This is our page serving function that generates web pages ---------------------------------------------------------------------
boolean sendMyPage(char* URL) {



  // Check if the requested URL matches "/"
  if (strcmp(URL, "/") == 0) {
    // Use WiServer's print and println functions to write out the page content
    WiServer.print("<html>");
    WiServer.print("Hello World!");
    WiServer.print("<p>");
    WiServer.print("Volume is FAT");
    WiServer.print(volume.fatType(), DEC);
    WiServer.print("</p>");
    WiServer.print("Files found in root:");
    WiServer.print("<p>");
    root.ls(LS_DATE | LS_SIZE);

    WiServer.print("</html>");
    ListFiles(0);
    // URL was recognized
  }
 
   URL++;
  if (file.open(&root, URL, O_READ)) { //this opens the file at the url that was requested if it exists

    }else{
    }
  int16_t n;
  uint8_t buf[3];// nothing special about 7, just a lucky number.
  while ((n = file.read(buf, sizeof(buf))) > 0) {
    for (uint8_t i = 0; i < n; i++) WiServer.print(buf[i]);
  }

  file.close();
  return true;
  // URL not found
  return false;
}

//--------------------------------------------------------------------------------------------------------------------------------
void ListFiles(uint8_t flags) {

  dir_t p;

  root.rewind();
  WiServer.println("<ul>");
  while (root.readDir(p) > 0) {
    // done if past last used entry
    if (p.name[0] == DIR_NAME_FREE) break;

    // skip deleted entry and entries for . and  ..
    if (p.name[0] == DIR_NAME_DELETED || p.name[0] == '.') continue;

    // only list subdirectories and files
    if (!DIR_IS_FILE_OR_SUBDIR(&p)) continue;

    // print any indent spaces
    WiServer.print("<li><a href=\"");
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        WiServer.print('.');
      }
      WiServer.print(p.name[i]);
    }
    WiServer.print("\">");

    // print file name with possible blank fill
    for (uint8_t i = 0; i < 11; i++) {
      if (p.name[i] == ' ') continue;
      if (i == 8) {
        WiServer.print('.');
      }
      WiServer.print(p.name[i]);
    }

    WiServer.print("</a>");

    if (DIR_IS_SUBDIR(&p)) {
      WiServer.print('/');
    }

    // print modify date/time if requested
    if (flags & LS_DATE) {
      root.printFatDate(p.lastWriteDate);
      WiServer.print(' ');
      root.printFatTime(p.lastWriteTime);
    }
    // print size if requested
    if (!DIR_IS_SUBDIR(&p) && (flags & LS_SIZE)) {
      WiServer.print(' ');
      WiServer.print(p.fileSize);
    }
    WiServer.println("</li>");
  }
  WiServer.println("</ul>");
}




void setup() {  //----------------------------------------------------------------------------------------------------------------
  Serial.begin(9600);
 
  pinMode(5, OUTPUT);
  pinMode(3, OUTPUT);
  WiServer.setIndicatorPins(5, 3);
  PgmPrint("Free RAM: ");
  Serial.println(FreeRam()); 

  pinMode(10, OUTPUT);                       // set the SS pin as an output (necessary!)
  digitalWrite(10, HIGH);

  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!card.init(SPI_HALF_SPEED, 7)) error("card.init failed!");

  // initialize a FAT volume
  if (!volume.init(&card)) error("vol.init failed!");

  PgmPrint("Volume is FAT");
  Serial.println(volume.fatType(),DEC);
  Serial.println();

  if (!root.openRoot(&volume)) error("openRoot failed");

  // list file in root with date and size
  PgmPrintln("Files found in root:");
  root.ls(LS_DATE | LS_SIZE);
  Serial.println();

  // Recursive list of all directories
  PgmPrintln("Files found in all dirs:");
  root.ls(LS_R);

  Serial.println();
  PgmPrintln("Done");

  WiServer.init(sendMyPage);
}



void loop() {  //-----------------------------------------------------------------------------------------------------------------
  WiServer.server_task();

}
Dino
 
Posts: 4
Joined: Thu Sep 02, 2010 6:41 pm

Re: WiServer serving larger files

Postby GregEigsti » Fri Sep 17, 2010 10:05 pm

There are a couple of problems here
1) You are running a webserver trying to serve "large" files on an Arduino - the AVR processor is pretty low powered and is never going to give super fantastic throughput.
2) You are using WiServer which has abstracted away much of the nitty gritty. You may realize better perf if you go with the WebServer example or a plain TCP app.

That said; if you are willing to study the uIP docs and play around you will be able to find perf gains. Unfortunately there is no way to increase your RAM other than stepping up to a more "powerful" processor - e.g. mega...

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: WiServer serving larger files

Postby Dino » Sat Sep 18, 2010 12:25 am

Ahh I found one reason why the files were loading slow. My antivirus was intercepting the data and taking FOREVER to scan everything. Made an exclusion and it's much faster now. :D

Still doesn't fix my low RAM... Oh well
Dino
 
Posts: 4
Joined: Thu Sep 02, 2010 6:41 pm

Re: WiServer serving larger files

Postby GregEigsti » Sat Sep 18, 2010 1:05 am

You might look at the mega; some folks around here have had luck getting the WiShield to work with the mega - but its not direct plug and play - there is definitely some tweaking involved to make it work. I have never tried it - but the mega does have ~4X more RAM than the Duemilanove (with 328). Definitely worth a go if you need more... And besides the mega has a ton more IO - which is what interests me. I have just never had the need - yet... The Arduino site is plastered with "we are cooking up something new" - might mean something cool!

Or you could get real good at writing AVR asm code :lol:

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: WiServer serving larger files

Postby mengineer » Sat Sep 18, 2010 7:22 pm

The only Arduino I've used is the Mega and I'm happy with it, and only a few file edits to get the WiShield working(I don't have it stacked like a shield though,just makes more since to use wire/headers and have it side by side in my enclosure with all the inputs I'm using). I'm not loading any large web pages. Just mainly posting data to pachube and running a 4D Systems 2.4" uOLED Touch screen with a decent amount of graphics(although the display has its own processor,which rocks with the pre-built functions for everything) and it's pretty quick, with a bunch of sensors,RFID,etc. Been on the edge of buying a Leaf Labs Maple for a little bit now, seeing as it uses the same programmer as Arduino more less,ARM processor, and the WiShield woks, but kind wanted to wait to see how much the bigger version of the maple is when it's complete. How ever, since Arduino's site says "were cooking up something new" as Greg mentioned I'm a little anxious to see what it is. Personally I'm kind of hoping for something a little more powerful like the Maple, with the WiFi built in would be even better. Back on topic though, if you decide to go with a Mega send me a message and I'll send you all the files edited already with data flash working at the same time as WiFi.
"Every day's a new adventure, you gotta take what you can!"
mengineer
 
Posts: 22
Joined: Mon Apr 12, 2010 8:49 pm
Location: Buffalo, NY

Re: WiServer serving larger files

Postby GregEigsti » Sun Sep 19, 2010 7:38 pm

Want a mega so bad! Was just about to order one with the latest batch of parts but... I'm going to hold off a bit and wait and see what the new Arduino stuff is all about first. Would hate to order some new boards and have them immediately replaced with something much cooler!

If and when I get a mega I'll definately be hitting you up for help with getting the WiShield running on it! Have a project in mind (automated indoor garden) that would benefit from all of the IO.

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: WiServer serving larger files

Postby mengineer » Mon Sep 20, 2010 12:08 pm

Another gardener, so I'm not a loser!LOL. Started wit gram as a kid outside and a few years ago turned into a few hundred $'s of lights,nutes,and media. If you've heard of Urban Garden Magazine, you can email and ask if they have any open testers position and do some research for them,learn a little and get free stuff. Last thing I tested was a Botanicare Complete drip setup a few months before it hit market, grew a lil,learned a lil(first time with hydro,soil guy with nothing but organic nutes), and ende up with a $400ish system for free. Anyways, I'm done running this thread completely off topic, originally just wanted to leave some input on mega. But has anyone checked MicroChip's site lately for the ZeroG module, http://www.microchip.com/stellent/idcplg?IdcService=SS_GET_PAGE&nodeId=2884. It's been replaced with their own version, MRF24WB0MA built-in PCB antenna, MRF24WB0MB supports external antennas. Looks same, specs seem same, hope all is the same/identical, so the Async code library doesn't change. Maybe this is why they haven't had much in stock lately for all those who have been asking.
"Every day's a new adventure, you gotta take what you can!"
mengineer
 
Posts: 22
Joined: Mon Apr 12, 2010 8:49 pm
Location: Buffalo, NY

Re: WiServer serving larger files

Postby GregEigsti » Mon Sep 20, 2010 6:20 pm

Good info - thanks! One of these days I'll get to my garden - and instrument the heck out of it ;)

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: WiServer serving larger files

Postby bdodds » Mon Nov 15, 2010 11:31 pm

So has anyone figured out how to serve larger files from an SD card over the WiShield yet?
I need to serve up some jpg files in the 500 to 600k range...
bdodds
 
Posts: 20
Joined: Tue Oct 19, 2010 6:01 pm
Location: Orange County CA


Return to WiServer

Who is online

Users browsing this forum: No registered users and 2 guests