HTTP GET with WebClient

Discussion about any of the included sketches (i.e. WebServer), or user-generated applications.

HTTP GET with WebClient

Postby parkerkrhoyt » Sun Nov 29, 2009 10:41 am

I've modified the WebClient.pde code to perform what I believe is an HTTP GET request. The sketch compiles and loads, and the LED on the WiShield board activates, but so far as I can tell, the request is never made. The destination of the request is a server script that increments a number in a file every time the page is viewed. The number never increments however, which leads me to believe the request is never made. It looks like there are some "twitter" string dependencies in the webclient.c file, but when I change the variables, I get compilation errors. I don't know enough C to figure out how to debug this, so any pointers would be greatly appreciated.

If you want to see the server script: http://www.kevinhoyt.org/wishield.cfm
If you want to see the file increment: http://www.kevinhoyt.org/wishield.txt

Code: Select all
#include <WiShield.h>

#define WIRELESS_MODE_INFRA   1
#define WIRELESS_MODE_ADHOC   2

// Wireless configuration parameters
unsigned char local_ip[] = {10,0,1,5};           // IP address of WiShield
unsigned char gateway_ip[] = {192,168,15,1};   // Router or gateway IP address
unsigned char subnet_mask[] = {255,255,255,0};   // Subnet mask for the local network
const prog_char ssid[] PROGMEM = {"MySSID"};      // Maximum 32 bytes

// 0 - open
// 1 - WEP
// 2 - WPA
// 3 - WPA2
unsigned char security_type = 3;

// WPA/WPA2 passphrase
// Maximum 64 characters
const prog_char security_passphrase[] PROGMEM = {"MyPassPhrase"};   // max 64 characters

// WEP 128-bit keys
prog_uchar wep_keys[] PROGMEM = {   0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d,   // Key 0
                           0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   0x00,   // Key 1
                           0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   0x00,   // Key 2
                           0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   0x00   // Key 3
                        };

// Setup the wireless mode
// Infrastructure - Connect to AP
// AdHoc - Connect to another WiFi device
unsigned char wireless_mode = WIRELESS_MODE_INFRA;

unsigned char ssid_len;
unsigned char security_passphrase_len;

// Setup
void setup()
{
   WiFi.init();
}

// Loop count
unsigned char loop_cnt = 0;

// Web site IP
char site_ip[] = {208,112,3,152};

// Web site HTTP GET request
const prog_char twitter[] PROGMEM = {"GET /wishield.cfm HTTP/1.1\r\nHost: www.kevinhoyt.org\r\n"};

// Loop construct
void loop()
{
   // Send the request on first iteration
   if( loop_cnt == 0 )
   {
      webclient_get( site_ip, 80, "/" );
      loop_cnt = 1;
   }
   
   WiFi.run();
}
parkerkrhoyt
 
Posts: 8
Joined: Fri Nov 27, 2009 7:55 pm

Re: HTTP GET with WebClient

Postby GregEigsti » Sun Nov 29, 2009 4:39 pm

It looks like WebClient.pde is tailored towards twitter. Have you taken a look at the SimpleClient example? Might be an easier (less twitter) place to get started.

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: HTTP GET with WebClient

Postby parkerkrhoyt » Mon Nov 30, 2009 11:58 am

SimpleClient relies on WiServer, and I have a ATMEGA168 chip, so I can't run WiServer. Based on WebClient however, it looks like HTTP requests can be made without relying on WiServer, so I was hoping to avoid the upgrade cost. SimpleClient does look really slick though. Guess I'll have to upgrade after all.

Still open to tips on modifying WebClient if anybody has them!

Thanks!
parkerkrhoyt
 
Posts: 8
Joined: Fri Nov 27, 2009 7:55 pm

Re: HTTP GET with WebClient

Postby gsxrex » Mon Nov 30, 2009 2:59 pm

Let me try and take a look at it later tonight. You just want to do a standard get of the first URL you posted, which must be listening to incoming connections, and then writes the txt file that you can view from the second link?

My guess is that something in the string is malformed, so the server is probably rejecting us (but that's just a guess right now).
User avatar
gsxrex
 
Posts: 115
Joined: Thu Apr 30, 2009 9:49 am

Re: HTTP GET with WebClient

Postby GregEigsti » Mon Nov 30, 2009 10:17 pm

parkerkrhoyt,
First off, thank you for providing the links to your server script and "incremented variable page"; these were extremely helpful in reproducing your problem and finding a fix for you. Yes, I did say a fix ;)

I refactored the WebClient code - removing the "twitter" goop and making it more web page generic. I also added a timer to the code to time when the "GET" events would happen - anyone who has looked at my sample scripts knows that timers/ISRs are a fetish of mine ;)

First off, I think that the problem was due to one (and potentially another) issue.
1) Your "twitter get string" contained a single "\r\n" (newline) at the end of the string. I thought this looked a little off because I was pretty sure that you needed to have two newlines (e.g. "\r\n\r\n") at the end of the string. I proved this to be true using telnet to send commands to your server. If you want more details on the use of telnet to figure out the situation just let me know - telnet can be very helpful to figure out just what you need to do on the network before trying to do so via the WiShield ;)
2) I have personally run into an issue where if you call a WiShield "send/connect" type command (in your case webclient_get()) too fast after calling WiFi.init() in setup() the WiShield is not yet completely initialized and sometimes the "send/connect" will not work. The original code had the webclient_get() happening immediately the first time that loop() was executed. This is where I made use of the timer to back the send off to only occurring once every 10 seconds. Again this is sample code and you may not want to send every 10 seconds - but what I have done is show that this sample sketch will work. Just speak up to get some help massaging the code to do what you want it to do - or better yet get the C tutorials out and start working on becoming an expert ;)

Also, since you are running a 168 the MsTimer2 may not fit; there are a multitude of ways to back off the first send; this was just an easy way for me to do it.

WebClient.pde
Code: Select all
/*
* Web Client
*
* A simple web client example using the WiShield 1.0
*/

#include <WiShield.h>
#include <MsTimer2.h>

#define WIRELESS_MODE_INFRA   1
#define WIRELESS_MODE_ADHOC   2

// Wireless configuration parameters ----------------------------------------
unsigned char local_ip[] = {192,168,1,2};   // IP address of WiShield
unsigned char gateway_ip[] = {192,168,1,1};   // router or gateway IP address
unsigned char subnet_mask[] = {255,255,255,0};   // subnet mask for the local network
const prog_char ssid[] PROGMEM = {"ASYNCLABS"};      // max 32 bytes

unsigned char security_type = 0;   // 0 - open; 1 - WEP; 2 - WPA; 3 - WPA2

// WPA/WPA2 passphrase
const prog_char security_passphrase[] PROGMEM = {"12345678"};   // max 64 characters

// WEP 128-bit keys
// sample HEX keys
prog_uchar wep_keys[] PROGMEM = {   0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d,   // Key 0
                           0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   0x00,   // Key 1
                           0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   0x00,   // Key 2
                           0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   0x00   // Key 3
                        };

// setup the wireless mode
// infrastructure - connect to AP
// adhoc - connect to another WiFi device
unsigned char wireless_mode = WIRELESS_MODE_INFRA;

unsigned char ssid_len;
unsigned char security_passphrase_len;
//---------------------------------------------------------------------------

volatile boolean intTimer;

//MsTimer2 one minute timer ISR
void timerISR() {
   intTimer = true;
}

void setup()
{
   Serial.begin(9600);
   
   WiFi.init();
   
   intTimer = false;
   //setup the timerISR to hit the server every 10 seconds
   MsTimer2::set(10000, timerISR);
   MsTimer2::start();   
}

// The stack does not have support for DNS and therefore cannot resolve
// host names. It needs actual IP addresses of the server.
char server_ip[] = {208,112,3,152};

// This string can be used to send a request to your server to get the specified "page"
const prog_char webpage[] PROGMEM = {"GET /wishield.cfm HTTP/1.1\r\nHost: www.kevinhoyt.org\r\n\r\n"};

void loop()
{
   if (true == intTimer) {
      Serial.println("Hitting the server...");
      intTimer = false;
      webclient_get(server_ip, 80, "/");
   }

   WiFi.run();
}


webclient.c - unchanged except for "de-twitter-ifying"
Code: Select all
/******************************************************************************

  Filename:      webclient.c
  Description:   Webclient app for the WiShield 1.0

******************************************************************************

  TCP/IP stack and driver for the WiShield 1.0 wireless devices

  Copyright(c) 2009 Async Labs Inc. All rights reserved.

  This program is free software; you can redistribute it and/or modify it
  under the terms of version 2 of the GNU General Public License as
  published by the Free Software Foundation.

  This program is distributed in the hope that it will be useful, but WITHOUT
  ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
  FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
  more details.

  You should have received a copy of the GNU General Public License along with
  this program; if not, write to the Free Software Foundation, Inc., 59
  Temple Place - Suite 330, Boston, MA  02111-1307, USA.

  Contact Information:
  <[email protected]>

   Author               Date        Comment
  ---------------------------------------------------------------
   AsyncLabs         05/29/2009   Initial version

*****************************************************************************/

#include "uip.h"
#include "uiplib.h"
#include "webclient.h"
#include "config.h"

#include <string.h>

#define WEBCLIENT_TIMEOUT 100

#define WEBCLIENT_STATE_STATUSLINE 0
#define WEBCLIENT_STATE_HEADERS    1
#define WEBCLIENT_STATE_DATA       2
#define WEBCLIENT_STATE_CLOSE      3

#define HTTPFLAG_NONE   0
#define HTTPFLAG_OK     1
#define HTTPFLAG_MOVED  2
#define HTTPFLAG_ERROR  3


#define ISO_nl       0x0a
#define ISO_cr       0x0d

static struct webclient_state s;

const char http_11[9] =
   /* "HTTP/1.1" */
{0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0 };
const char http_200[5] =
   /* "200 " */
{0x32, 0x30, 0x30, 0x20, 0 };
const char http_301[5] =
   /* "301 " */
{0x33, 0x30, 0x31, 0x20, 0 };
const char http_302[5] =
   /* "302 " */
{0x33, 0x30, 0x32, 0x20, 0 };

/*-----------------------------------------------------------------------------------*/
char* webclient_mimetype(void)
{
   return s.mimetype;
}
/*-----------------------------------------------------------------------------------*/
char* webclient_filename(void)
{
   return s.file;
}
/*-----------------------------------------------------------------------------------*/
char* webclient_hostname(void)
{
   return s.host;
}
/*-----------------------------------------------------------------------------------*/
unsigned short webclient_port(void)
{
   return s.port;
}
/*-----------------------------------------------------------------------------------*/
void webclient_init(void)
{

}
/*-----------------------------------------------------------------------------------*/
static void init_connection(void)
{
   s.state = WEBCLIENT_STATE_STATUSLINE;

   // set the length of the client string to be transmitted
   s.getrequestleft = strlen_P(webpage);

   s.getrequestptr = 0;

   s.httpheaderlineptr = 0;
}
/*-----------------------------------------------------------------------------------*/
void webclient_close(void)
{
   s.state = WEBCLIENT_STATE_CLOSE;
}
/*-----------------------------------------------------------------------------------*/
unsigned char webclient_get(char *host, u16_t port, char *file)
{
   struct uip_conn *conn;
   uip_ipaddr_t ipaddr;

   uip_ipaddr(&ipaddr, host[0],host[1],host[2],host[3]);

   conn = uip_connect(&ipaddr, htons(port));

   if(conn == NULL) {
      return 0;
   }

   s.port = port;
   strncpy(s.file, file, sizeof(s.file));
   strncpy(s.host, host, sizeof(s.host));

   init_connection();
   return 1;
}
/*-----------------------------------------------------------------------------------*/
static unsigned char* copy_string(unsigned char *dest, const unsigned char *src, unsigned char len)
{
   strncpy(dest, src, len);
   return dest + len;
}
/*-----------------------------------------------------------------------------------*/
static void senddata(void)
{
   u16_t len;
   char *getrequest;
   char *cptr;

   if(s.getrequestleft > 0) {
      cptr = getrequest = (char *)uip_appdata;

#if 0
      cptr = copy_string(cptr, http_get, sizeof(http_get) - 1);
      cptr = copy_string(cptr, s.file, strlen(s.file));
      *cptr++ = ISO_space;
      cptr = copy_string(cptr, http_10, sizeof(http_10) - 1);

      cptr = copy_string(cptr, http_crnl, sizeof(http_crnl) - 1);

      cptr = copy_string(cptr, http_host, sizeof(http_host) - 1);
      cptr = copy_string(cptr, s.host, strlen(s.host));
      cptr = copy_string(cptr, http_crnl, sizeof(http_crnl) - 1);

      cptr = copy_string(cptr, http_user_agent_fields,
            strlen(http_user_agent_fields));
#endif

      // copy the client transmit string into the TX buffer
      memcpy_P(cptr, webpage, strlen_P(webpage));

      len = s.getrequestleft > uip_mss()?uip_mss():s.getrequestleft;
      uip_send(&(getrequest[s.getrequestptr]), len);
   }
}
/*-----------------------------------------------------------------------------------*/
static void acked(void)
{
   u16_t len;

   if(s.getrequestleft > 0) {
      len = s.getrequestleft > uip_mss()?uip_mss():s.getrequestleft;
      s.getrequestleft -= len;
      s.getrequestptr += len;
   }
}
/*-----------------------------------------------------------------------------------*/
static u16_t parse_statusline(u16_t len)
{
   char *cptr;
   char* temp;

   while(len > 0 && s.httpheaderlineptr < sizeof(s.httpheaderline)) {
      s.httpheaderline[s.httpheaderlineptr] = *(char *)uip_appdata;
      //((char *)uip_appdata)++;
      temp = (char *)uip_appdata;
      temp++;
      uip_appdata = temp;
      --len;
      if(s.httpheaderline[s.httpheaderlineptr] == ISO_nl) {

         if((strncmp(s.httpheaderline, http_11,sizeof(http_11) - 1) == 0)) {
            cptr = &(s.httpheaderline[9]);
            s.httpflag = HTTPFLAG_NONE;
            if(strncmp(cptr, http_200, sizeof(http_200) - 1) == 0) {
               /* 200 OK */
               s.httpflag = HTTPFLAG_OK;
            }
            else if(strncmp(cptr, http_301, sizeof(http_301) - 1) == 0 ||
                  strncmp(cptr, http_302, sizeof(http_302) - 1) == 0) {
               /* 301 Moved permanently or 302 Found. Location: header line
                * will contain the new location. */
               s.httpflag = HTTPFLAG_MOVED;
            }
            else {
               s.httpheaderline[s.httpheaderlineptr - 1] = 0;
            }
         }
         else {
            uip_abort();
            webclient_aborted();
            return 0;
         }

         /* We're done parsing the status line, so we reset the pointer
          * and start parsing the HTTP headers.*/
         s.httpheaderlineptr = 0;
         s.state = WEBCLIENT_STATE_HEADERS;
         break;
      }
      else {
         ++s.httpheaderlineptr;
      }
   }
   return len;
}
/*-----------------------------------------------------------------------------------*/
static u16_t parse_headers(u16_t len)
{
   char *cptr;
   static unsigned char i;
   char* temp;

   while(len > 0 && s.httpheaderlineptr < sizeof(s.httpheaderline)) {
      s.httpheaderline[s.httpheaderlineptr] = *(char *)uip_appdata;
      //++((char *)uip_appdata);
      temp = (char *)uip_appdata;
      temp++;
      uip_appdata = temp;
      --len;
      if(s.httpheaderline[s.httpheaderlineptr] == ISO_nl) {
         /* We have an entire HTTP header line in s.httpheaderline, so
          * we parse it. */
         if(s.httpheaderline[0] == ISO_cr) {
            /* This was the last header line (i.e., and empty "\r\n"), so
             * we are done with the headers and proceed with the actual
             * data. */
            s.state = WEBCLIENT_STATE_DATA;
            return len;
         }

         /* We're done parsing, so we reset the pointer and start the
          * next line. */
         s.httpheaderlineptr = 0;
      }
      else {
         ++s.httpheaderlineptr;
      }
   }

   return len;
}
/*-----------------------------------------------------------------------------------*/
static void newdata(void)
{
   u16_t len;

   len = uip_datalen();

   if(s.state == WEBCLIENT_STATE_STATUSLINE) {
      len = parse_statusline(len);
   }

   if(s.state == WEBCLIENT_STATE_HEADERS && len > 0) {
      len = parse_headers(len);
   }

   if(len > 0 && s.state == WEBCLIENT_STATE_DATA &&
         s.httpflag != HTTPFLAG_MOVED) {
      webclient_datahandler((char *)uip_appdata, len);
   }
}
/*-----------------------------------------------------------------------------------*/
void webclient_appcall(void)
{
   if(uip_connected()) {
      s.timer = 0;
      s.state = WEBCLIENT_STATE_STATUSLINE;
      senddata();
      webclient_connected();
      return;
   }

   if(s.state == WEBCLIENT_STATE_CLOSE) {
      webclient_closed();
      uip_abort();
      return;
   }

   if(uip_aborted()) {
      webclient_aborted();
   }

   if(uip_timedout()) {
      webclient_timedout();
   }

   if(uip_acked()) {
      s.timer = 0;
      acked();
   }

   if(uip_newdata()) {
      s.timer = 0;
      newdata();
   }

   if(uip_rexmit() || uip_newdata() || uip_acked()) {
      senddata();
   }
   else if(uip_poll()) {
      ++s.timer;
      if(s.timer == WEBCLIENT_TIMEOUT) {
         webclient_timedout();
         uip_abort();
         return;
      }
   }

   if(uip_closed()) {
      if(s.httpflag != HTTPFLAG_MOVED) {
         /* Send NULL data to signal EOF. */
         webclient_datahandler(NULL, 0);
      }
      else {
         //webclient_get(s.host, s.port, s.file);
      }
   }
}
/*-----------------------------------------------------------------------------------*/

void webclient_datahandler(char *data, u16_t len)
{
}

void webclient_connected(void)
{
}

void webclient_timedout(void)
{
}

void webclient_aborted(void)
{
}

void webclient_closed(void)
{
}

/** @} */
/** @} */


Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: HTTP GET with WebClient

Postby parkerkrhoyt » Fri Dec 18, 2009 2:13 pm

Whoa!

I was expecting, like most public forums, that I'd post my question only to sit there idle. Nay, here is a pile of code for me! Woo hoo! Thanks!

Having said that, I uploaded the sketch (the timer fit) and the card seemed to be connecting to my wireless network (indicated via the LED), but still no ping against that server - no increment. Now it's entirely likely that I'm not putting the right addresses in the right spot, so I'll have to fiddle with that.

However, since the WiShield doesn't support DHCP and needs a fixed IP, does that mean I need to change anything on my wireless router? Can everything in general still be DHCP, but when the WiShield goes to connect it uses a specific IP? Assuming that IP is open, the DHCP system will let it have it? Or do I need to configure the router differently? Any keywords I should look for?

Thanks again for the code and support!
Kevin
parkerkrhoyt
 
Posts: 8
Joined: Fri Nov 27, 2009 7:55 pm

Re: HTTP GET with WebClient

Postby GregEigsti » Fri Dec 18, 2009 2:42 pm

Generally you are safe using an unused DHCP address - but for how long? If some other DHCP capable device comes online and the DHCP server gives that device your WiShield's address then there will be problems. What I have done on my DHCP server is to tell it to hand out addresses between, say, 192.168.0.100 and 192.168.0.200 - this way I know that any address under 192.168.0.100 will never be used by DHCP devices and are therefore safe to use as "static" addresses. There are probably better ways to achieve this but it works for me (so far).

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: HTTP GET with WebClient

Postby parkerkrhoyt » Fri Dec 18, 2009 3:04 pm

Ah!

Good to know. I'll try some reconfiguration and change up the IP in the sketch. I look forward to reporting the success of my first WiShield project (ye olde humidor).

Thanks again for the help!
Kevin
parkerkrhoyt
 
Posts: 8
Joined: Fri Nov 27, 2009 7:55 pm

Re: HTTP GET with WebClient

Postby parkerkrhoyt » Mon Dec 21, 2009 10:28 pm

As a quick follow-up as I progress ...

It seems that the code above would fit on a 168, but the stack would dump within minutes and the connection severed. Changed up to a 328 and have been running the ping program for hours with no problems whatsoever. Very excited! Now to wire in the temperature and humidity sensors and introduce the board to my humidor.

Well follow-up when I get further along.

Thanks,
Kevin
parkerkrhoyt
 
Posts: 8
Joined: Fri Nov 27, 2009 7:55 pm

Re: HTTP GET with WebClient

Postby parkerkrhoyt » Tue Dec 22, 2009 8:34 am

Scratch that ...

Turns out to be that while there's enough space to hold the sketch, there's not enough space to hold much beyond the basic HTTP response. This is something to be aware of especially if you might get an error response from the HTTP call. Usually error responses are pretty verbose. Eat up enough space with that reply (error or otherwise) and you run out of memory. Poof there goes your stack.

Kevin
parkerkrhoyt
 
Posts: 8
Joined: Fri Nov 27, 2009 7:55 pm

Next

Return to Sketches and Applications

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron