syntax.us Let the syntax do the talking
Blog Contact Posts Questions Tags Hire Me

You are here: Linux101 > Shell 101 > shell101_wget

Linux Shell 101

Question:
In Shell101 how to wget Yahoo CSV?

To do Machine Learning on Linux I need to wget some data from Yahoo.

The data I like is a CSV file full of prices for the S&P 500 going back to 1950.

I used Emacs to build a simple Bash script to wget the data I want:
ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ cat wget_gspc.bash
#!/bin/bash

# ~ann/wget_gspc.bash

# This script should wget a CSV file from Yahoo.

# I should cd to the right folder:

cd ~ann

# If wget is not in my path, I might need to use apt-get to install wget.

# Here is the syntax to install wget from the root account:
# apt-get install wget

# If wget is installed and if I can connect to Yahoo,
# the syntax below should give me a new CSV file:

export TKRH='%5EGSPC'
export TKR='GSPC'
rm -f ${TKR}.csv
wget --output-document=${TKR}.csv  http://ichart.finance.yahoo.com/table.csv?s=${TKRH} 

date
ls -la ${TKR}.csv

exit
ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ 


I ran the script and I saw this:
ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ ll wget_gspc.bash
-rw-rw-r-- 1 ann ann 570 Feb 20 01:37 wget_gspc.bash
ann@feb ~ $ 
ann@feb ~ $ chmod 754 wget_gspc.bash
ann@feb ~ $ 
ann@feb ~ $ ll wget_gspc.bash
-rwxr-xr-- 1 ann ann 570 Feb 20 01:37 wget_gspc.bash
ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ ./wget_gspc.bash 
--2015-02-20 01:39:15--  http://ichart.finance.yahoo.com/table.csv?s=%5EGSPC
Resolving ichart.finance.yahoo.com (ichart.finance.yahoo.com)... 206.190.61.106, 206.190.61.107
Connecting to ichart.finance.yahoo.com (ichart.finance.yahoo.com)|206.190.61.106|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/csv]
Saving to: ‘GSPC.csv’

    [  <=>                                  ] 895,559     2.33MB/s   in 0.4s   

2015-02-20 01:39:16 (2.33 MB/s) - ‘GSPC.csv’ saved [895559]

Fri Feb 20 01:39:16 UTC 2015
-rw-rw-r-- 1 ann ann 895559 Feb 20 01:39 GSPC.csv
ann@feb ~ $ 
ann@feb ~ $ 


ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ head GSPC.csv 
Date,Open,High,Low,Close,Volume,Adj Close
2015-02-19,2099.25,2102.13,2090.79,2097.45,3247100000,2097.45
2015-02-18,2099.16,2100.23,2092.15,2099.68,3370020000,2099.68
2015-02-17,2096.47,2101.30,2089.80,2100.34,3361750000,2100.34
2015-02-13,2088.78,2097.03,2086.70,2096.99,3527450000,2096.99
2015-02-12,2069.98,2088.53,2069.98,2088.48,3788350000,2088.48
2015-02-11,2068.55,2073.48,2057.99,2068.53,3596860000,2068.53
2015-02-10,2049.38,2070.86,2048.62,2068.59,3669850000,2068.59
2015-02-09,2053.47,2056.16,2041.88,2046.74,3549540000,2046.74
2015-02-06,2062.28,2072.40,2049.97,2055.47,4232970000,2055.47
ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ 


ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ tail GSPC.csv
1950-01-16,16.72,16.72,16.72,16.72,1460000,16.72
1950-01-13,16.67,16.67,16.67,16.67,3330000,16.67
1950-01-12,16.76,16.76,16.76,16.76,2970000,16.76
1950-01-11,17.09,17.09,17.09,17.09,2630000,17.09
1950-01-10,17.03,17.03,17.03,17.03,2160000,17.03
1950-01-09,17.08,17.08,17.08,17.08,2520000,17.08
1950-01-06,16.98,16.98,16.98,16.98,2010000,16.98
1950-01-05,16.93,16.93,16.93,16.93,2550000,16.93
1950-01-04,16.85,16.85,16.85,16.85,1890000,16.85
1950-01-03,16.66,16.66,16.66,16.66,1260000,16.66
ann@feb ~ $ 
ann@feb ~ $ 
ann@feb ~ $ 
So, that is the third shell programming demo of shell101.



You are here: Linux101 > Shell 101 > shell101_wget
You can ask questions in Dan's Machine Learning Class Forum:
https://groups.google.com/forum/#!forum/dan101

syntax.us Let the syntax do the talking
Blog Contact Posts Questions Tags Hire Me