Source changes to allow simultaneous WiFi and dataflash

Postings related to the second version of the WiShield

Source changes to allow simultaneous WiFi and dataflash

Postby CapnBry » Wed Sep 08, 2010 6:05 am

In the Dataflash on WiShield 2.0 thread I learned that some people may be having a hard time getting the dataflash and WiFi to simultaneously work. I had some brief issues with this I fixed by referencing the datasheet as well as some knowledge of how the SPI bus operates, so I thought I'd share the information. I use the dataflash chip to store static web content and serve it over the wifi, with the device running for several days without any issues so I can say it is pretty stable over that time, but I have not had a chance to test a real long term (48+ days).

Step 1: Update the dataflash source code
I pulled the source for the dataflash library from the asynclabs github. I could read the manufacturer's ID, but that was about it. After several failed attempts to read and write anything to flash, I pulled up the datasheet. It appeared that the source used old deprecated versions of some of the opcodes and the status register. These changes needed to be made to dataflash.h
Code: Select all
-#define FlashPageRead         0x52   // Main memory page read
+#define FlashPageRead         0xD2   // Main memory page read
-#define Buf1Read         0x54   // Buffer 1 read
+#define Buf1Read         0xD4   // Buffer 1 read
-#define Buf2Read         0x56   // Buffer 2 read
-#define StatusReg         0x57   // Status register
+#define Buf2Read         0xD6   // Buffer 2 read
+#define StatusReg         0xD7   // Status register
-#define ContArrayRead         0x68   // Continuous Array Read (Note : Only A/B-parts supported)
+#define ContArrayRead         0x03   // Continuous Array Read (Note : Only A/B-parts supported) low freq

Another issue is that the class assumes that the static variables PageBits and PageSize are initialized before calling anything that reads or writes to flash memory (buffer memory was ok). These are only set when reading the status, so that meant that the first of every flash read / write would have undefined results. To alleviate this I moved a call to Read_DF_Status() to the init routine.

Cont_Flash_Read_Enable() really didn't do anything, because there was no Cont_Read() function. Continuous read is like streaming data directly from flash, each byte you read increments the address pointer, which will roll from one page to the next; it is a pretty useful read method, so I changed Cont_Flash_Read_Enable() to use the low-frequency opcode and stop discarding the first four bytes of data, as well as added a Cont_Flash_Read() to get the streamed data.

Finally I added Sizes() function for returning the size of the page address bits and page size, for use in an uploader program I wrote that page aligned the start of each file.

Step 2: Understanding the SPI bus
I could read and write to flash and buffer all I wanted now, but using it with WiFi caused the Arduino to stop responding to web requests as soon as it started to serve the first flash file. The issue here is simple. The SPI bus is basically a chain of devices who all share the same clock and rx/tx lines but each device on the bus has its own unique line that is called Slave Select (SS#). When this signal is high, the device ignores all data on the bus. When this line is low, the device is "selected" to be the recipient of the SPI bus data. What I noticed is that when you use the dataflash, it enables the DF's slaveselect, does its business, and returns, leaving the slave line low. When the WiFi tries to send the data you're pulling from the flash, the data it is sending is now going to both the dataflash and wifi, which means both of them can respond too. This of course is a problem considering you've got two devices with different communication protocols sending and receiving at the same time with no arbitration. The solution is to enable the dataflash chip when you're using it and disable it using once you've done your business.

Sample Code!
Serving a static page from dataflash.
Code: Select all

const char FNAME000[] PROGMEM = "favicon.ico";
const char FNAME001[] PROGMEM = "fire.png";
const char FNAME002[] PROGMEM = "";

const struct flash_file {
  const char *fname;
  const unsigned int page;
  const unsigned int size;
} FLASHFILES[] PROGMEM = {
  { FNAME000,   3, 1625 },
  { FNAME001,   7, 11824 },
  { FNAME002,  30, 4218 },
  { 0, 0, 0},
};

boolean sendPage(char* URL)
{
  ++URL;  // WARNING: URL no longer has leading '/'

  const struct flash_file *file = FLASHFILES;
  while (pgm_read_word(&file->fname))
  {
    if (strcmp_P(URL, (const prog_char *)pgm_read_word(&file->fname)) == 0)
    {
      sendFlashFile(file);
      return true;
    }
    ++file;
  }


void sendFlashFile(const struct flash_file *file)
{
  unsigned int size = pgm_read_word(&file->size);
 
  dflash.Cont_Flash_Read_Enable(pgm_read_word(&file->page), 0);
  while (size-- > 0)
    WiServer.write(dflash.Cont_Flash_Read());
  dflash.DF_CS_inactive();
}

Note that this is largely inefficient for files over ~350 bytes because the whole file is read from flash every time even though the WiFi can only transfer 400 at a time. It works though.

But your comment about needing to 'unselect' the dataflash chip on the SPI bus makes me wonder if you've also put a DF_CS_inactive() call at the end of the dflash.Cont_Flash_Read() method. Like just inside Cont_Flash_Read() so it will release the dataflash chip just as you said. Is this the case?

Actually no. The WiServer.write() method only writes the byte to the buffer, not actually out the device. If you toggle the slave select during the continuous read operation, the dataflash chip will stop sending because it is expecting you to send a command. When the sendPage() returns, the wiserver will activate the wifi and send the data you've written with write(), which is ok because you've deactivated the SS# (called CS for some reason in the code) before returning from your sendPage() method.

My edited version of dataflash I use is available from my website.
CapnBry
 
Posts: 10
Joined: Wed Aug 18, 2010 11:21 am

Re: Source changes to allow simultaneous WiFi and dataflash

Postby spidermonkey04 » Wed Sep 08, 2010 12:22 pm

I believe we touched this subject in the Dataflash forum, and should probably post some updated code. I've been using the Dataflash for a while, so it's definitely an asset to the shield.

BTW, the address bits will always be 10 and page size always 528 (unless you reflash to use 512), so these two could be put in a #define to save codespace.

---Jared
spidermonkey04
 
Posts: 66
Joined: Thu Oct 29, 2009 6:45 pm

Re: Source changes to allow simultaneous WiFi and dataflash

Postby CapnBry » Wed Sep 08, 2010 1:37 pm

Oh man I guess I should have looked there first.

Actually the Sizes() method gets left out by the linker so it's OK. I only use it in the upload sketch when it connects so the client app could work with any flash configuration. In my main application it only reads using the continuous array read so page size and address bits don't really concern it. The header describing the file table is output by the upload program, and has that #define like you've suggested, but I don't think I use it anywhere.

I'd like to at some point update the file serve function to not always read/write the complete file every time it is called, like I see you've done too. However, that's a pretty low priority for me considering how many other hardware/software/web content changes I'd still like to make :D

EDIT: I think the function gets linked out. I think I tested by looking at the assembly but now it has been a month or so and I forget.
CapnBry
 
Posts: 10
Joined: Wed Aug 18, 2010 11:21 am

Re: Source changes to allow simultaneous WiFi and dataflash

Postby CapnBry » Wed Sep 08, 2010 1:43 pm

Oh I see what you're saying about #defining the page size and address bits, from your other post in the dataflash forum. Yeah I guess that would works too. I could certainly use more PROGMEM space, considering I'm down to about a KB and I've been very careful about my code.
CapnBry
 
Posts: 10
Joined: Wed Aug 18, 2010 11:21 am

Re: Source changes to allow simultaneous WiFi and dataflash

Postby comperem » Sat Sep 11, 2010 3:54 am

Jared,

Your posts are what I've been keeping an eye on for flash functionality simultaneously with the wifi but last I saw you were "gonna post something soon...". Guess I kept looking in the wrong area.

Have you achieved similar functionality and also posted example code? I notice you mention page sizes and address bits. That sounds to me like you're writing specific amounts of data to specific locations in the flash memory similar to a memcpy(). CapnBry is reading the entire file (or memory region) to completion which, like he mentioned is a start but not perhaps the ideal functionality. The type of API I'm more interested in will be writing and reading specific amounts to and from specific memory locations for data acquisition purposes.

BTW, do either of you guys have a feel for the time is takes for the flash to read and write N bytes? Are we talking microseconds or milliseconds to write a 2-byte integer??? It takes about 112 microseconds to do a single A/D call on the duemilanove. Combining this with an estimate on time to write a 2-byte integer will yield an estimated maximum data acquisition rate while storing data in the flash.

Marc
comperem
 
Posts: 12
Joined: Mon Jul 13, 2009 6:34 am

Re: Source changes to allow simultaneous WiFi and dataflash

Postby CapnBry » Mon Sep 13, 2010 7:01 am

Actually, I find that my functionality is more ideal than a simple memcpy()-like routine. If you wanted a read memcpy() that emulates a flat memory model it would just look like this
Code: Select all
void *memcpy_F(void *dest, const void *src, size_t num)
{
  void *retVal = dest;
  uint16_t page = (uint16_t)src / PageSize;
  uint16_t offset = (uint16_t)src % PageSize;
  dflash.Cont_Flash_Read_Enable(page, offset);
  while (num-- > 0)
    *(uint8_t *)dest++ = dflash.Cont_Flash_Read();
  dflash.DF_CS_inactive();
  return retVal;
}

Note how the code is the same, except I've precalculated the page and size information and associated a file name with it so the flash area is more like a file system than a chunk of memory.

Now writing on the other hand, is more complicated. There's only one write function, Buffer_To_Page. This replaces the entire page with the buffer, so you can't just write randomly. If you're only going to write 2 bytes, then you'd Page_To_Buffer, Write_To_Buffer(your 2 bytes), and Buffer_To_Page to commit it. Obviously if you're thinking of writing 2 bytes every couple of milliseconds, you probably can't keep up. In this case I'd recommend writing a whole buffer full of 2-byte values and commit the whole page to flash. Of course if you want to be able to read it before it is committed, your memcpy_F would have to detect if it needs to read from buffer instead of flash or commit the page before doing a read.

As far as speed goes, I haven't done any measurements.
CapnBry
 
Posts: 10
Joined: Wed Aug 18, 2010 11:21 am

Re: Source changes to allow simultaneous WiFi and dataflash

Postby spidermonkey04 » Wed Sep 15, 2010 2:23 pm

I kinda stopped posting about this stuff because nobody seemed interested. I do have some functions for reading/writing certain number of bytes from/to specific location. I'll post them up tomorrow when I get more time. The main hurdle there is keeping track of where the NEXT read/write will be

The Adr/Page bits I speak of are chip specific. Our 16Mbit chips hold 528 bytes per page with 4096 pages total. In order to count that high (4095) 10 bits are needed. I proposed to hard-code these values because they won't change unless you swap in a new chip, say 64Mbit, which has more total pages and I think double the bytes per page.

Not sure on speed of a single byte, but last I tried I was getting 7-8Kb/s.

---Jared
spidermonkey04
 
Posts: 66
Joined: Thu Oct 29, 2009 6:45 pm

Re: Source changes to allow simultaneous WiFi and dataflash

Postby spidermonkey04 » Thu Sep 16, 2010 1:30 pm

I guess I already posted the function I was thinkin about with the webpage stuff, which prob doesn't really apply.

I did a few speed tests on writing to a buffer, just for grins.

Single byte: 28 microseconds
Float (4bytes): 32 microseconds
Fill Buffer 1 and commit to page: 13.6 milliseconds
Continuous read a page: 5.4 milliseconds
Continuous read 10 pages: 14.2 milliseconds

Like CapnBry said, you'd be better off filling up a buffer, then writing the buffer to page. Remember there are 2 buffers on this chip. So when Buffer 1 is full, you could commit that to a page while still writing to Buffer 2.

If you really need speed, I tried using Direct port manipulation on the cable select pin instead of using the painfully slow digitalWrite(). Now writing a float to buffer only takes 20 microseconds!
Code: Select all
void Dataflash::DF_CS_inactive()
{
  //digitalWrite(FLASHSEL,HIGH);
  PORTD |= B10000000;  // Directly set Dig7 HIGH
}
void Dataflash::DF_CS_active()
{
  //digitalWrite(FLASHSEL,LOW);
  PORTD &= B01111111;  // Directly set Dig7 LOW
}

It looks like you've got it figured out though, good luck!

---Jared
spidermonkey04
 
Posts: 66
Joined: Thu Oct 29, 2009 6:45 pm

Re: Source changes to allow simultaneous WiFi and dataflash

Postby CapnBry » Fri Sep 17, 2010 6:07 am

Speaking of speed, I modified my sendFlashFile() function last night to only send the 310 bytes that the UIP buffer will be accepting. I think spidermonkey04 did something like this as well in one of his posts which I can't seem to find now. On the first send, this writes 19 more bytes than it has to but that's ok with me considering the slightly lower complexity.
Code: Select all

#define HTTP_HEADER_LENGTH 19 // "HTTP/1.0 200 OK\r\n\r\n"
void sendFlashFile(const struct flash_file_t *file)
{
  // Note we mess with the underlying UIP stack to prevent reading the entire
  // file each time from flash just to discard all but 300 bytes of it
  // Speeds up an 11kB send by approximately 3x (up to 1.5KB/sec)
  uip_tcp_appstate_t *app = &(uip_conn->appstate);
  unsigned int sentBytes = app->ackedCount;

  // The first time through, the buffer contains the header but nothing is acked yet
  // so don't mess with any of the state, just send the first segment 
  if (app->ackedCount > 0)
  {
   app->cursor = (char *)sentBytes;
   sentBytes -= HTTP_HEADER_LENGTH;
  }

  unsigned int page = pgm_read_word(&file->page) + (sentBytes / DATAFLASH_PAGE_BYTES);
  unsigned int off = sentBytes % DATAFLASH_PAGE_BYTES;
  unsigned int size = pgm_read_word(&file->size);
  unsigned int sendSize = size - sentBytes;

  if (sendSize > UIP_TCP_MSS)
    sendSize = UIP_TCP_MSS;
   
  dflash.Cont_Flash_Read_Enable(page, off);
  while (sendSize-- > 0)
    WiServer.write(dflash.Cont_Flash_Read());
  dflash.DF_CS_inactive();
 
  // Pretend that we've sent the whole file
  app->cursor = (char *)(HTTP_HEADER_LENGTH + size);
}


Like it says, this can serve files up to 1.5KB/s, which is actually the theoretical maximum you can serve to a windows machine (windows 7 only?) due to their 200ms ack delay. You can only send 310 bytes of payload per ack delay, so 1550 bytes. I wish there were some way to indicate to the client that they should disable their delay because we don't have the memory to have more than one packet on the wire!
CapnBry
 
Posts: 10
Joined: Wed Aug 18, 2010 11:21 am

Re: Source changes to allow simultaneous WiFi and dataflash

Postby CapnBry » Fri Sep 17, 2010 7:04 am

Oh snap looks like you've already discovered uip-split. Did you ever get that enabled in the WiServer code? I hacked it quickly into stack.c, but I'm curious if you've already produced an optimal solution.

EDIT: Also because my quick hack doesn't work, I'm still seeing 329 bytes coming out at a time. Looks like the size of the send isn't qualifying for a split and if I force it to split, the second packet is garbage. I assume this is because part of the uip_data buffer is used for the WiFi header and the split doesn't take this into account
18:25:59.886257 IP 192.168.1.7.34986 > 192.168.1.252.80: Flags [P.], seq 1:120, ack 1, win 5840, length 119
18:25:59.894839 IP 192.168.1.252.80 > 192.168.1.7.34986: Flags [.], ack 120, win 329, length 0
18:26:00.892170 IP truncated-ip - 14 bytes missing! 192.168.1.252.80 > 192.168.1.7.34986: Flags [P.], seq 172:344, ack 120, win 329, length 172
CapnBry
 
Posts: 10
Joined: Wed Aug 18, 2010 11:21 am

Next

Return to WiShield 2.0

Who is online

Users browsing this forum: No registered users and 1 guest