dashwood.net -

Ryan Stefan's Micro Blog

Quick Collection of Python CSV Read/Write Techniques

Jun 142019

Import CSV as Dict

  • Creates ordered dict
  • You can increase file size limit
  • Using next() can bypass the header row
import csv

# Dict reader creates an ordered dict (first row will be headers)
with open('./data/file.csv', newline='') as file:
    # Huge csv files might give you a size limit error
    results = csv.DictReader(file, delimiter=';', quotechar='*', quoting=csv.QUOTE_ALL)

    # next() can help in iterations sometimes
    for row in results:
        # prints each item in the column with header 'key'

Import CSV with No Header (nested lists)

  • newline='' prevents blank lines
  • csv.reader uses indexes [0], [1] 
# newline='' prevents blank lines
with open('./data/file.csv', newline='') as file:
    results = csv.reader(file, delimiter=':', quoting=csv.QUOTE_NONE)
    for row in results:
        # csv reader uses indexes

Writing and Creating Headers

  • Create a csv.writer object
  • Create header manually before loop
  • Nested lists are better than tuples inside lists
  • writer.writerow and writer.writerows 
# Creates a csv writer object
writer = csv.writer(
    file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)

# Write header first if you would like
writer.writerow(['title', 'price', 'shipping'])

''' Tuples inside list (list inside lists are usually better though).
If you're using tuples and they are variable size, note that a single tuple
will convert to string type in a loop so indexing it [0] won't work. '''
products = [['slinky', '$5', 'Free'],
            ['pogo', '$12', '$6'],
            ['Yoyo', '$7', '$2']]

# write each row normal
for item in products:
    writer.writerow(map(str, item))

# Writes all items into a single row
writer.writerow(sum(products, []))

# Writes all 3 rows

Using DictWriter for Headers

  • fieldnames indicates header to object
  • writer.writeheader() writes those fields
# DictWriter field names will add the headers for you when you call writeheader()
with open("./data/file.csv", "w") as file:
    writer = csv.DictWriter(
        file, fieldnames=['title', 'price', 'shipping'],

    writer.writerows([['slinky', '$5', 'Free'],
                      ['pogo', '$12', '$6'],
                      ['Yoyo', '$7', '$2']])

Bonus - Flatten any List

  • Function will flatten any level of nested lists
  • or type == tuple() to catch tuples too
# -- Bonus (Off Topic) --
# You can flatten any list with type checking and recursion
l = [1, 2, [3, 4, [5, 6]], 7, 8, [9, [10]]]
output = []
def flatten_list(l):
    for i in l:
        if type(i) == list:


SQLAlchemy and Friends

Feb 242019
ALTER TABLE product_que ALTER COLUMN attempts TYPE integer USING attempts::integer;
ALTER TABLE product_que ALTER COLUMN amazon TYPE integer USING amazon::integer;
ALTER TABLE product_que ALTER COLUMN ebay TYPE integer USING ebay::integer;
ALTER TABLE product_que ALTER COLUMN etsy TYPE integer USING etsy::integer;
query = self.session.query(db.ProdQue).filter(or_(db.ProdQue.amazon > 0,
                                                          db.ProdQue.ebay > 0, db.ProdQue.etsy > 0)).limit(5000)