Web Scraping on Secured Sites that require UserName and Password.

12-30-03, 02:08 PM

John West

I have a Web Site that I want to monitor and gather
information from which requires a secured user id and
password of which I have but don't know how to gather get
by the secured log in to get to the pages I need. I have
gathered information from sites that don't require secured
login by using the following code:

Function readHtmlPage(ByVal url As String) As String
Dim objResponse As WebResponse
Dim objRequest As WebRequest
Dim result As String
objRequest = System.Net.HttpWebRequest.Create(url)
objResponse = objRequest.GetResponse()
Dim sr As New StreamReader
(objResponse.GetResponseStream())
result = sr.ReadToEnd()
sr.Close()
Return result
End Function

But don't know how to get around the sites that require
secured log-in such as https protocol.

Any help or direction would be appreciated.

12-30-03, 08:40 PM

Microsoft

Have you tried the Internet Transfer control? I use that ocassionally to get
FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
You'd just need to find a copy to install.

"John West" <johnwestgtown> wrote in message
news:280a
[..]

12-30-03, 10:58 PM

Herfried K. Wagner [MVP]

* "Microsoft" <twilson> scripsit:
> Have you tried the Internet Transfer control? I use that ocassionally to get
> FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
> Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
> You'd just need to find a copy to install.

For FTP:

<http://www.freevbcode.com/ShowCode.Asp?ID=4655>
<http://www.allapi.net/downloads/NetFTP.zip>
<http://www.visualbuilder.com/article/viewarticle.asp?id=1234>
<http://www.abderaware.com/> (FTP component)
<http://www.csharphelp.com/archives/archive9.html>
<http://www.codeproject.com/csharp/FTPDriver1.asp>
....

12-31-03, 09:11 AM

Cor

Hi Herfried,

Can we do the Pentagon also with your help.
Nobody was answering this thread, but there is always one
.........................
:-((

Cor

Similar Threads
Printer shares require username and password? I am having a problem with a simple XP pro (SP2)system (server) when I set the printer to share over the network on the client machine I can see the server but the printer...
Images require username and password? Hi All, If I open an email, why does it require me to enter my username and password in order for me to see images? I have already unchecked the Don't download pictures or...
Require username & password on home/small office network In a simple home/small office netowork using XP how do I require someone to enter the username and password of the computer they want to connect to when they navigate to that...
Prompting a username/password on a non-secured database Hello, I created a database that is password protected for some of the staff in my department. I've also created other non-secured databases that they use on a daily basis....
I can't go to any sites that require login and password I get this message every time "Your current security settings prohibit running ActiveX controls on this page. As a result, the page may not display correctly." Any time I...