Google Ads

February 20, 2018

Use Python objects to process and store csv file data

In Python data from csv files can be easily loaded and processed for analysis or automating other jobs. In this tutorial, we will see how data from a csv file can be stored easily in Python objects and spit out the processed data from an object as necessary. For this exercise, we assume we have a csv file containing temperature data for 28 days where the first column contains days records and the second column contains temperatures records. First we will load the csv file's temperatures data into an object and then show the data and temperature average from the object. After that, we find and store the differences in days temperature from the object to another object. Finally, we save the differences in days temperature data from the object to a csv file. A sample of the csv file contents is shown below:

1 27
2 25
3 23
4 30
5 19
6 20
7 13
8 30

Now, in Python first we will have to create an object for holding these data from the csv file. For this, in the Python object we need to have two lists, one for days records and the other one for temperatures records. Now, we define the Python object as below:

# define object for holding a set of temperature data
class TempData():
    def __init__(self):
        self.day = [] # list of days
        self.temp = [] # list of temperature

Now, we create two object instance variables of this type where one will hold temperatures data as in the csv file and the other one will hold temperature differences data.

csv_imported_data = TempData() # create object for holding csv file temperature data
processed_data = TempData() # create object for holding processed data

Now, we can open the csv file and load the csv columns data into the object lists for days and temperatures data.

csv_file = open('data.csv', 'r') # open the csv file for reading
reader = csv.reader(csv_file) # pass the file to csv reader object for extracting data

for row in reader: # get the row data from the csv file
    csv_imported_data.day.append(float(row[0])) # get the first column data
    csv_imported_data.temp.append(float(row[1])) # get the second column data

csv_file.close() # close the csv file

Now, lets show the temperature data from the object:

# print temperature data from the temperature object
for i in range(len(csv_imported_data.day)):
    print(csv_imported_data.day[i], csv_imported_data.temp[i])

Now, lets show the average temperature from the object's temperature data:

# show average temperature from the temperature object
print('Average temperature for', len(csv_imported_data.day), 'days was', sum(csv_imported_data.temp)/len(csv_imported_data.temp))

Now, lets find the temperature difference from the object's temperature data and store difference data into another object of same type:

# find difference between temperatures and store in temperature data object
for i in range(len(csv_imported_data.day)):
    if (i == 0): # no difference for first data
        processed_data.day.append(csv_imported_data.day[i])
        processed_data.temp.append(0)
    else:
        processed_data.day.append(abs(csv_imported_data.day[i] - csv_imported_data.day[i-1]))
        processed_data.temp.append(abs(csv_imported_data.temp[i] - csv_imported_data.temp[i-1]))

Now, we show the temperature difference data from the object:

# print temperature difference data from the temperature object
for i in range(len(processed_data.day)):
    print(processed_data.day[i], processed_data.temp[i])

Finally, we save the temperature difference data from the object to a new csv file so that we can retrieve the temperature difference data without running the Python script again:

csv_file = open('processed_temp_data.csv', 'w', newline='') # create a new csv file for writing
csv_writer = csv.writer(csv_file) # pass the file to csv writer object for writing data

for row in zip(processed_data.day, processed_data.temp): # create the row data from the the temperature object
    csv_writer.writerow(row)

csv_file.close()

Download the full example code and CSV files:


Cheers!
Imam

5 comments:

  1. Hi Imam, Great post. One question, Say I wanted to extend upon this. I have a folder with 12 temperature CSV_files for each month of 2017.

    January
    February
    March
    April
    May
    June
    July
    August
    September
    October
    November
    December

    To load these into python would I create a list of months like this:

    temperature_months = []

    How would I then load these CSV into the python object TempData()??

    ReplyDelete
    Replies
    1. Hi Ben, pleased that the post was somewhat useful. If we want to load data from several files all we need a list which will contain TempData() objects. For example, first we can create an empty list for the objects like this:

      temperature_months = []

      and then as we read data from each file we just have to insert object to the list and use list index notation to add data to the object like following:

      # create object and insert into the object list
      temperature_months.append(TempData())

      # access object from object list and insert data into the object
      temperature_months[0].day.append(csv_imported_data.day[i])
      temperature_months[0].temp.append(0)

      Delete
  2. Also if you make the comment entry widow larger it will make it easier to enter new comments. At the moment it only shows three lines. Cheers

    ReplyDelete