Python Programming/Internet

Web Page Harvesting Networks The urllib module which is bundled with python can be used for web interaction. This module provides a file-like interface for web urls. Getting page text as a string An example of reading the contents of a webpage import urllib.request as urllib pageText = urllib.urlopen("http://www.spam.org/eggs.html").read() print(pageText) Processing page text line by line: import urllib.request as urllib for line in urllib.urlopen("https://en.wikibooks.org/wiki/Python_Programming/Internet"): print(line) Get and post methods can be used, too. import urllib.request as urllib params = urllib.urlencode({"plato":1, "socrates":10, "sophokles":4, "arkhimedes":11}) # Using GET method pageText = urllib.urlopen("http://international-philosophy.com/greece?%s" % params).read() print(pageText) # Using POST method pageText = urllib.urlopen("http://international-philosophy.com/greece", params).read() print(pageText) Downloading files To save the content of a page on the internet directly to a file, you can read() it and save it as a string to a file object import urllib2 data = urllib2.urlopen("http://upload.wikimedia.org/wikibooks/en/9/91/Python_Programming.pdf", "pythonbook.pdf").read() # not recommended as if you are downloading 1gb+ file, will store all data in ram. file = open('Python_Programming.pdf','wb') file.write(data) file.close() This will download the file from here and save it to a file "pythonbook.pdf" on your hard drive. Other functions The urllib module includes other functions that may be helpful when writing programs that use the internet: >>> plain_text = "This isn't suitable for putting in a URL" >>> print(urllib.quote(plain_text)) This%20isn%27t%20suitable%20for%20putting%20in%20a%20URL >>> print(urllib.quote_plus(plain_text)) This+isn%27t+suitable+for+putting+in+a+URL The urlencode function, described above converts a dictionary of key-value pairs into a query string to pass to a URL, the quote and quote_plus functions encode normal strings. The quote_plus function uses plus signs for spaces, for use in submitting data for form fields. The unquote and unquote_plus functions do the reverse, converting urlencoded text to plain text. Email With Python, MIME compatible emails can be sent. This requires an installed SMTP server. import smtplib from email.mime.text import MIMEText msg = MIMEText( """Hi there, This is a test email message. Greetings""") me = 'sender@example.com' you = 'receiver@example.com' msg['Subject'] = 'Hello!' msg['From'] = me msg['To'] = you s = smtplib.SMTP() s.connect() s.sendmail(me, [you], msg.as_string()) s.quit() This sends the sample message from 'sender@example.com' to 'receiver@example.com'. External links urllib.request, docs.python.org HOWTO Fetch Internet Resources Using The urllib Package, docs.python.org urllib2 for Python 2, docs.python.org HOWTO Fetch Internet Resources Using urllib2 — Python 2.7, docs.python.org Web Page Harvesting Networks