dashwood.net -

Ryan Stefan's Micro Blog

Python Dict Key Recursive Attempt Function

Jun 132019

So I'm dealing with this huge database where each item has a bunch of levels and some have different keys than others. So I made a script that takes a list of lists and tries those keys, if they work the function breaks. Pretty simple and I'm sure I could make it cleaner, but it works well enough and I don't expect any 4 level items any time soon. Anyways, here's the code:

for item in items:
''' items is a list of dicts to run the keys on '''
results = {} def key_attempter(name, key_lists): nonlocal item nonlocal results for key in key_lists: if len(key) == 1: try: results[name] = item[key[0]] break except: pass elif len(key) == 2: try: results[name] = item[key[0]][key[1]] break except: pass elif len(key) == 3: try: results[name] = item[key[0]][key[1]][key[2]] break except: pass feat_lists = { 'price': [ ['hdpData', 'homeInfo', 'price'], ['price'], ['hdpData', 'priceForHDP'], ['priceLabel'], ['hdpData', 'homeInfo', 'zestimate'], ['hdpData', 'festimate'] ], 'bed': [ ['beds'], ['hdpData', 'homeInfo', 'bedrooms'] ], 'bath': [ ['baths'], ['hdpData', 'homeInfo', 'bathrooms'] ]} for k in feat_lists.keys(): key_attempter(k, feat_lists[k])

return results

Scout Spider for Finding Fresh Proxy Websites

Jun 122019

With so many proxy website URLs all over the place, it's difficult to tell which one's actually have new proxies posted or if you're just receiving the same old proxies that are cluttering up your list and wasting time on testing. So, I wrote a spider that will scrape proxies off of URLs and compare the first 15 results to see how different the results are. Easy peasy.

I omitted the spider settings, Request func, and the callback func to keep it compact:

from scrapy import Spider
from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher
from difflib import SequenceMatcher

import threading
import re
import csv

IPPortPatternGlobal = re.compile(
    r'(?P<ip>(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?))'  # noqa
    r'(?=.*?(?:(?:(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?))|(?P<port>\d{2,5})))',  # noqa

file_name = 'scout_results'
lock = threading.Lock()
threads = []
pdata = {}

with open(f"./data/{file_name}.csv") as file:
        results = csv.DictReader(file, delimiter=',')
        for row in results:
                if int(row["count"]) > 0:
                    pdata[row['url']] = {'first_15': row['first_15'], 'count': row['count']}
            except Exception as e:
                print(f'Error: {e}')

class SingleSpider(Spider):
    def __init__(self):
        dispatcher.connect(self.spider_closed, signals.spider_closed)
        global file_name
        self.new_pdata = open(f"./data/{file_name}.csv", "w+")

    def thread_compare(self, data):
        with lock:
            global pdata
            url = data[0].strip()
            f_15 = str(data[1]).strip()
            count = str(data[2]).strip()
                ip_diff = str(self.compare(f_15, pdata[url]['first_15']))
                count_diff = str(abs(int(count) - int(pdata[url]['count'])))
                print(f'{url} - ip: {ip_diff} count: {count_diff}')
            except Exception as e:
                ip_diff = 'empty'
                count_diff = 'empty'
                print(f'Nothing to compare: {e}')


    def compare(block1, block2):
        s = SequenceMatcher(lambda x: x in "\n", block1, block2)
        return s.quick_ratio()

    def spider_closed(self, spider):

Multithreading and Run Once Decorator

Jun 112019

I've been coding again and just remembered how well this website works for keeping track of cool tricks I learn. Sometimes it's really hard to find simple and generic examples of things to help teach the fundamentals. I needed to write to a file without opening the text document 1000 times and I finally found a really clean example that helped me understand the pieces.

Edit** Threadpool is a lot easier and you can thread inside a loop:

from multiprocessing.pool import ThreadPool as Pool

threads = 100

p = Pool(threads)
p.map(function, list)

More complicated version:

import threading
lock = threading.Lock()
def thread_test(num):
    phrase = "I am number " + str(num)
    with lock:
        print phrase
        f.write(phrase + "\n")
threads = []
f = open("text.txt", 'w')
for i in range (100):
    t = threading.Thread(target = thread_test, args = (i,))
while threading.activeCount() > 1:

Close something on Scrapy spider close without using a pipeline:

from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher

class MySpider(CrawlSpider):
    def __init__(self):
        dispatcher.connect(self.spider_closed, signals.spider_closed)

    def spider_closed(self, spider):
      # second param is instance of spder about to be closed.

Instead of using an if time or if count to activate something I found a decorator that will make sure the function on runs once:

def run_once(f):
    def wrapper(*args, **kwargs):
        if not wrapper.has_run:
            wrapper.has_run = True
            return f(*args, **kwargs)
    wrapper.has_run = False
    return wrapper

def my_function(foo, bar):
    return foo+bar

You can also resize the terminal inside the code:

import sys
sys.stdout.write("\x1b[8;{rows};{cols}t".format(rows=46, cols=54))

I got stuck for a while trying to get my repository to let me login without creating an ssh key (super annoying imo) and I figured out that I added the ssh url for the origin url and needed to reset it to the http:

change origin url
git remote set-url origin <url-with-your-username>

Combine mp3 files with linux:

ls *.mp3
sudo apt-get install mp3wrap
mp3wrap output.mp3 *.mp3

Regex is always better than splitting a bunch of times and making the code messy. Plus it's a lot easier to pick up the code later on and figure out what's going on. So I decided to take my regex to the next level and start labeling groups (I'm even going to give it it's very own tag :3:

pat = r'(?<=\,\"searchResults\"\:\{)(?<list_results>.*)(?=\,\"resultsHash\"\:)'

m = re.match(pat, url)
if m:
    self.domain = m.group('list_results')

Using Deep Truth to Maintain a Positive Focus - Powerful Positive Thinking

May 282019

Affirmations have always been a really effective way to intentionally change my mental state regardless of my external influences. I’m a pretty overly sensitive person (I used to say I was tweaked wrong) and I have a lot of trouble keeping myself balanced. I even used to have constant panic attacks, sometimes multiple in a week, which made my life a constant struggle that only I could see. I really needed a way to fight back and gain some control over my own mental state. I chose to dive into philosophy with all of my being and I didn’t find out until much later that the part of the brain that lights up during a panic attack is the same part of the brain that lights up when contemplating the infinite or “oneness”, thank you Alan Watts. I haven’t had a panic attack or any anxiety in years.

About Affirmations

The general idea behind an affirmation is that if you repeat something enough, your mind will incorporate it into your personality. While I was searching for ways to control my mental state, I realized that affirmations were a lot more effective if they were something that I actually believed was true. After this realization, I set out to find the perfect set of affirmations that were both true (at least, for me) and maximally empowering. Once I did this, affirmations became a way to guide my focus on positive powerful thinking rather than trying to force myself to believe something by simple repetition. Now when I repeat these affirmations, it’s like wanting to buy something and not having enough money, but then you remember you had a $20 bill stashed in your pants pocket. “Oh yeah now I remember, and everything is fine.” The real trick with my affirmations, besides actually finding the best ones to use, is understanding them to such a deep level that there is no doubt to their validity.

I'll briefly explain the philosophy behind my affirmations, but the most important thing to focus on is the emotional response behind each one.

What truth will cause you to feel the deepest gratitude in your life?

What idea will fill you with wonder for the future?

My 4 Affirmations of Deep Truth

  1. I am always on the right path
  2. There is always something wonderful happening
  3. My love and value is always making an impact
  4. Reality is both responsive and limitless

1. Purpose

I am always on the right path

The first mantra has been the same since their inception and does a really good job of setting the frame for the rest of them. It’s true because you were born with the neural network you had and reacted to the first thing you interacted with and have been reacting ever since. Therefore, you’ve been on this path since the beginning and there was never any other path. In a negative light this might feel like the universe is random and that there is no real point to existence, but in a positive light, you imagined yourself into existence and you meticulously chose this path because it offered the most entertaining experience to watch. On top of that, it can set up the view that your entire life leads up to this moment of meditation where you had this epiphany and saw the reason behind all of the past struggle.

2. Gratitude and Growth

There is always something wonderful happening

The second one has changed a bit over time. The previous edition was “something wonderful is always coming”, but this creates a state of anticipation and you become the hopeful rather than the grateful. I’ve since realized that not only is there almost always something wonderful in my own life happening at any given moment, but there are always wonderful things happening all over the planet and it helps reinforce the fact that our reality is a friendly and wonderful place.

3. Self Confidence and Love

My love and value is always making an impact

This one has been really tough to nail down, but I knew it needed to reinforce the knowledge that I was making progress with my efforts regardless of how it seemed on the surface. Whether it be wasting time on a failed relationship or sinking tons of effort into a failed business venture, I always showed someone love with my efforts and learned from my struggles, making waves into the universe. Eventually these waves combine and come back into my life to help me overcome something I haven’t even considered in my future endeavors.

4. Possibility and Conviction

Reality is both responsive and limitless

It’s really easy to believe that the physical laws that surround our lives are the only laws that affect us, but the real truth behind reality is virtually unimaginable. I wanted a reminder that the reality we live in is an infinite collection of moments that respond to physical geometry (ours starts with the big bang) and that our thoughts can tap into this geometric pathway on which these moments combine into consciousness (time as we know it) and manifest changes that defy observable physical laws.


I don’t think there is really a wrong way to do affirmations, but I do think just repeating the words in your head isn’t very effective. I think that it’s very important to understand why each one is true. When I do them, I get in a meditative state somewhere quiet and I take my time with each one. I’ll repeat the affirmation and start thinking about why it is true. Once I remember why it’s completely true, I start to sink into a state of resonance with that belief. I really feel the emotion behind it and apply it to all the things that have been happening in my life that I’d been dealing with or worrying about. “Oh yeah now I remember, and everything is fine.”

I feel the relief of knowing I’m on the right path and that I didn’t screw my life up and go down an unrecoverable path. I feel the joy of all the wonderful things that have been happening that I was too busy to appreciate. I feel the love hiding behind all of the guilt I built up from fighting or spiteful behavior. I feel the satisfaction of seeing my efforts making an impact on my friends and on my own wisdom. I feel the wonder surge through me as I consider the possibilities my future holds in an infinite reality.

SQLAlchemy and Friends

Feb 242019
ALTER TABLE product_que ALTER COLUMN attempts TYPE integer USING attempts::integer;
ALTER TABLE product_que ALTER COLUMN amazon TYPE integer USING amazon::integer;
ALTER TABLE product_que ALTER COLUMN ebay TYPE integer USING ebay::integer;
ALTER TABLE product_que ALTER COLUMN etsy TYPE integer USING etsy::integer;
query = self.session.query(db.ProdQue).filter(or_(db.ProdQue.amazon > 0,
                                                          db.ProdQue.ebay > 0, db.ProdQue.etsy > 0)).limit(5000)