HTTP GET with WebClient

Discussion about any of the included sketches (i.e. WebServer), or user-generated applications.

Re: HTTP GET with WebClient

Postby Anthony » Wed Jan 13, 2010 7:39 pm

GregEigsti wrote:...I refactored the WebClient code - removing the "twitter" goop and making it more web page generic. ...

Thank you Greg so much for these two files, but I am not sure how I would modify them to work with my server.

One problem I have is that my server does not have a static IP address. However, it changes so rarely, that I don't mind hard-coding it's current IP address for now. I just hope someday we get DNS. Also, it's not the only domain on that IP address (I think that is called "virtual hosting"). Will your above code still work?
Anyway, I have put a temporary test server up at http://deepwaveenergy.com/arduino/adder.cgi?5,800,10
It's an adder, it just adds numbers together. The C++ source is:

Code: Select all
/////////////////////////////////////
//
// This is a cgi program that runs on a server waiting to add up numbers.
// To activate it, hit it's URL and put numbers (separated by commas) after a "?" like this:
//
//    http://deepwaveenergy.com/arduino/adder.cgi?5,800,10
//
//  It should return:
//  +--------------------------+
//  | Content-type: text/html  |
//  |                          |
//  | <html>                   |
//  | <body>                   |
//  | sum = 815<br>            |
//  | </body>                  |
//  | </html>                  |
//  +--------------------------+
//
#include <iostream>
#include <fstream>
#include <string>
#include <cstdio>

using namespace std;

int main( int argc, char* argv[], char* envp[] )
  {
  cout << "Content-type: text/html" << endl << endl;
  cout << "<html>" << endl;
  cout << "<body>" << endl;

  // Get the stuff past the "?" in the URL.
  char*  str = getenv( "QUERY_STRING" );
  int val = 0;
  int sum = 0;
  while( str != NULL  &&  *str != '\0' )
    {
    if( *str >= '0'  &&  *str <= '9' )
      val = (val * 10) + ((int)(*str) - (int)'0');
    else
      {
      sum += val;
      val = 0;
      }
    str++;
    }
  sum += val;

  cout << "sum = " << sum << "<br>" << endl;
  cout << "</body>" << endl;
  cout << "</html>" << endl;
  return 0;
  }


I figure my "const prog_char webpage[] PROGMEM should be:
Code: Select all
GET /arduino/adder.cgi?5,800,10 uIP/1.0
Host: deepwaveenergy.com




As you suggested, I tried: telnet 66.147.240.154 80
and successfully got back
Code: Select all
HTTP/1.1 200 OK
Date: Thu, 14 Jan 2010 01:58:42 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8l DAV/2 mod_auth_passthrough/2.1 FrontPage/5.0.2.2635
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

<html>
<body>
sum = 815<br>
</body>
</html>


I'm using a Boarduino (328), so I should have enough memory to do this, right? Other than changing the usual stuff (local_ip, gateway_ip, subnet, ssid, etc.) and PROGMEM, what else do I need to change? If I can get my Arduino to add 2 numbers together using this server, then I will be able to do what I really want to do.
Greg, you did such a great job helping Kevin; if you don't mind, can you help me too?
Anthony
 
Posts: 3
Joined: Wed Jan 13, 2010 11:25 am

Re: HTTP GET with WebClient

Postby GregEigsti » Thu Jan 14, 2010 1:31 am

Sure Anthony we'll give it a go ;)
You'll just have to forgive me for being really tired/stressed - the paying job is being a real pain :roll:

>> just hope someday we get DNS.
I have DNS working on the WiShield but to use it you need to tweak the WiShield 'driver code' to allow UDP and TCP (if you are using TCP, DNS requires UDP). Let me know if you want the goop. I have not posted the DNS sketch because I have yet to come up with a clean way to enable TCP and UDP for all app types that the WiShield supports.

>>Also, it's not the only domain on that IP address (I think that is called "virtual hosting")
Just make sure you include the 'Host:' portion in your GET request; the 'virtual domain' stuff that I am used to uses the Host part of the header to redirect to the proper web site.

>>I'm using a Boarduino (328), so I should have enough memory to do this, right?
This should be fine; most of my large sketches use only half of the 32K on the Duemilanove.

>>what else do I need to change?
Well depending upon what WiShield sample sketch that you are using that can vary. When your sketch does the GET some data will be returned and made available to you; so you will want to parse/validate and then make use of the returned number (think atoi() or scanf() to convert the ascii number to an int or whatever data type you want).

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: HTTP GET with WebClient

Postby Anthony » Thu Jan 14, 2010 10:30 am

Thank you Greg. I understand being busy and I promise I will not bug you until I'm absolutely stuck.
I'm starting with the code you posted in the start of this thread (Sketches and Applications -> HTTP GET with WebClient) on Nov 30. I'm going to try it for the first time this weekend. Once I get it working, I will post the code so that others can do use it. Otherwise, my next post will be a cry for help.
Say tuned
Anthony
 
Posts: 3
Joined: Wed Jan 13, 2010 11:25 am

Re: HTTP GET with WebClient

Postby GregEigsti » Sat Jan 16, 2010 8:56 am

Good luck!
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: HTTP GET with WebClient

Postby rcraddock » Sat Oct 16, 2010 1:43 pm

Hello All,

I've had an Aduino & WiShield for a while now and while I've managed to do some of the things I've wanted to I've come to a brick wall on some of them. One of these walls concerns this sketch, I've been trying to remove the webpage from PROGMEM so I can change it and embed variables in it programmatically.

If I'm way off here (in that I don't need to remove it from PROGMEM) then please correct me. If I do then some pointers on how to do so would be greatly appreciated! I've found the definition in config.h but I'm unsure what to change it to - all in all I'm very new to the C side of Arduino and have run out of luck reading and searching!

Many Thanks,

Richard
rcraddock
 
Posts: 4
Joined: Sat Oct 16, 2010 8:38 am

Re: HTTP GET with WebClient

Postby GregEigsti » Sat Oct 16, 2010 2:44 pm

Yes you will need to remove it from progmem as it is effectively read only there. Take a look at the C string handling functions particularly sprintf().

Greg
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: HTTP GET with WebClient

Postby rcraddock » Sun Oct 17, 2010 8:08 am

Hi Greg,

Thanks for the pointers. I'm still stuck on moving the webpage from PROGMEM. I've changed
Code: Select all
extern const prog_char webpage[];
to
Code: Select all
extern char webpage[];
in WebClient.c and
Code: Select all
const prog_char webpage[] PROGMEM = {"GET /wishield.cfm HTTP/1.1\r\nHost: www.kevinhoyt.org\r\n\r\n"};
to
Code: Select all
char webpage[] PROGMEM = {"GET /wishield.cfm HTTP/1.1\r\nHost: www.kevinhoyt.org\r\n\r\n"};


The sketch complies fine, however I no longer see the device accessing the page in the Apache logs (I did when it was still in progmem). I added some debug code to the webclient_connected section in WebClient.c and can tell it thinks it is connected but I'm struggling to know where to go!

It looks like webpage is simply not getting to the senddata function but I'm not sure why. I'll carry on trying to figure it out but if anyone knows if I've missed anything then I'd love to hear back.

Thanks for the pointers!

Richard
rcraddock
 
Posts: 4
Joined: Sat Oct 16, 2010 8:38 am

Re: HTTP GET with WebClient

Postby GregEigsti » Sun Oct 17, 2010 8:17 am

You missed a "PROGMEM" on that last line that you posted.
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

Re: HTTP GET with WebClient

Postby rcraddock » Sun Oct 17, 2010 8:21 am

Ops! That was (honestly) a mistake when posting, not when typing in the Arduino environment. The read line reads
Code: Select all
char webpage[] = {"GET /data.php HTTP/1.1\r\nHOST: www.website.com\r\n\r\n"};


Sorry to waste time and cause confusion, I'm using a laptop with Arduino and desktop to research.

I forgot to mention I also removed
Code: Select all
extern const prog_char webpage[];
from config.h and put
Code: Select all
extern char webpage[];
in WebClient.c.

Thanks,

Richard
rcraddock
 
Posts: 4
Joined: Sat Oct 16, 2010 8:38 am

Re: HTTP GET with WebClient

Postby GregEigsti » Sun Oct 17, 2010 8:37 am

In the original sample the name of the buffer to be sent is "twitter" - this is the char array that you are moving out of PROGMEM correct? (You probably have renamed it but its the buffer that contains your HTTP GET line - correct?) You also need to make a few other changes since stuff in PROGMEM is accessed differently than a normal array of chars. If you go to webclient.c and search for that buffer (e.g. twitter or whatever you have named it) you will find that it is accessed with some functions that end in _P - these are the special versions of those functions that operate on data in PROGMEM. So for instance
Code: Select all
// Originally this
memcpy_P(cptr, twitter, strlen_P(twitter));
// Change to
memcpy(cptr, twitter, strlen(twitter));
Check out the wiki!
uIP Stack Docs
Compatible Access Point List
WiShield user contrib branch - DNS, DHCP, AP Scanning, bug fixes, etc.
SlackLab.org - My geek projects blog.
User avatar
GregEigsti
 
Posts: 1067
Joined: Sun Aug 02, 2009 5:23 pm
Location: Sammamish WA USA (near Seattle)
  • Website

PreviousNext

Return to Sketches and Applications

Who is online

Users browsing this forum: No registered users and 1 guest