Fix isort in vscode for python2

Go to the vscode Python extension folder:

cd ~/.vscode/extensions/ms-python.python-2019.9.34911

Install latest version that is known to work with python 2:

pip2 install -t ./pythonFiles/lib/python --upgrade isort==4.3.17

python: Django-like testrunner

Django-like testrunner for non-django projects:

#!/usr/bin/env python

import argparse
import collections
import unittest
import sys


def main():
    parser = argparse.ArgumentParser(description='Django-like testrunner')
    parser.add_argument('specific_test', metavar='', type=str, nargs='?', help='Specifc test')
    parser.add_argument("--failfast", action="store_true")
    parser.add_argument("--verbosity", type=int, default=1)

    args = parser.parse_args()

    loader = unittest.TestLoader()
    all_tests = loader.discover('.', top_level_dir="./")
    suite = unittest.TestSuite()

    if args.specific_test:

        def walk_tests(tests):
            if isinstance(tests, collections.Iterable):
                for item in tests:
                    walk_tests(item)
                return
            if tests.id().startswith(args.specific_test):
                suite.addTest(tests)
            elif not str(tests).startswith("test"):
                sys.exit("Error in file %s" % tests)

        walk_tests(all_tests)
    else:
        suite.addTests(all_tests)

    unittest.TextTestRunner(verbosity=args.verbosity, failfast=args.failfast).run(suite)


if __name__ == "__main__":
    main()

List data samples for faker

faker is a very useful library for generating beautiful test data. However, I was not able to find a proper description with all providers. All samples (plus some useless data) can be printed and inspected with the following code:

from faker import Factory as FakerFactory

faker = FakerFactory.create()
methods = [method for method in dir(faker) if callable(getattr(faker, method)) and not method.startswith("_")]
for item in methods:
    try:
        print u"- %s:" % item
        print u"  %s" % getattr(faker, item)()
    except TypeError:
        pass

The output is:

- address:
  25611 Senger Cove
Cadenmouth, HI 46932-8013
- am_pm:
  AM
- boolean:
  False
- bothify:
  07 sb
- bs:
  re-contextualize customized mindshare
- building_number:
  86338
- catch_phrase:
  Expanded uniform complexity
- century:
  III
- chrome:
  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/5361 (KHTML, like Gecko) Chrome/14.0.879.0 Safari/5361
- city:
  Sporershire
- city_prefix:
  Lake
- city_suffix:
  port
- color_name:
  Indigo
- company:
  Flatley Inc
- company_email:
  kbaumbach@mckenzie.com
- company_suffix:
  LLC
- country:
  Syrian Arab Republic
- country_code:
  FI
- credit_card_expire:
  12/26
- credit_card_full:
  
Diners Club / Carte Blanche
Pierre Donnelly
36417005868777  04/26
CVC: 568
- credit_card_number:
  6011478274887340
- credit_card_provider:
  VISA 16 digit
- credit_card_security_code:
  081
- currency_code:
  RSD
- date:
  1982-04-04
- date_time:
  1999-05-19 00:50:24
- date_time_ad:
  1864-03-03 15:27:13
- date_time_between:
  2002-08-21 16:01:56
- date_time_between_dates:
  2017-03-08 21:56:07
- date_time_this_century:
  2016-01-14 03:30:51
- date_time_this_decade:
  2014-08-17 00:29:28
- date_time_this_month:
  2017-03-03 08:27:52
- date_time_this_year:
  2017-01-10 12:08:41
- day_of_month:
  04
- day_of_week:
  Saturday
- domain_name:
  altenwerth.com
- domain_word:
  cronawalker
- ean:
  3265654672776
- ean13:
  4119002606106
- ean8:
  13383411
- email:
  columbus64@gmail.com
- file_extension:
  png
- file_name:
  aperiam.html
- firefox:
  Mozilla/5.0 (Windows NT 6.1; it-IT; rv:1.9.2.20) Gecko/2012-07-23 16:34:01 Firefox/4.0
- first_name:
  Tella
- first_name_female:
  Oliva
- first_name_male:
  Arba
- format:
- free_email:
  milford.rempel@yahoo.com
- free_email_domain:
  yahoo.com
- geo_coordinate:
  -82.910152
- get_formatter:
- get_providers:
  [<faker.providers.user_agent.Provider object at 0x7fcce4201410>, <faker.providers.ssn.en_US.Provider object at 0x7fcce4201450>, <faker.providers.python.Provider object at 0x7fcce42013d0>, <faker.providers.profile.Provider object at 0x7fcce4201390>, <faker.providers.phone_number.en_US.Provider object at 0x7fcce4201350>, <faker.providers.person.en_US.Provider object at 0x7fcce4201310>, <faker.providers.misc.Provider object at 0x7fcce4201250>, <faker.providers.lorem.la.Provider object at 0x7fcce42012d0>, <faker.providers.job.en_US.Provider object at 0x7fcce4201290>, <faker.providers.internet.en_US.Provider object at 0x7fcce4201210>, <faker.providers.file.Provider object at 0x7fcce42011d0>, <faker.providers.date_time.Provider object at 0x7fcce4201190>, <faker.providers.currency.Provider object at 0x7fcce4201150>, <faker.providers.credit_card.Provider object at 0x7fcce4201110>, <faker.providers.company.en_US.Provider object at 0x7fcce4201090>, <faker.providers.color.en_US.Provider object at 0x7fcce42010d0>, <faker.providers.barcode.Provider object at 0x7fcce4201050>, <faker.providers.address.en_US.Provider object at 0x7fcce41e9fd0>]
- hex_color:
  #69d7c0
- image_url:
  http://www.lorempixum.com/373/972
- internet_explorer:
  Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.01; Trident/3.1)
- ipv4:
  129.146.124.124
- ipv6:
  ef5e:860d:c643:8233:0ff0:d40d:7739:9844
- iso8601:
  2007-01-25T19:03:46
- job:
  Dancer
- language_code:
  de
- last_name:
  Romaguera
- last_name_female:
  Heidenreich
- last_name_male:
  Bogan
- latitude:
  62.9329675
- lexify:
  UGTW
- linux_platform_token:
  X11; Linux x86_64
- linux_processor:
  x86_64
- locale:
  ru_TO
- longitude:
  -49.789527
- mac_address:
  41:9e:ad:ca:2a:fe
- mac_platform_token:
  Macintosh; PPC Mac OS X 10_5_2
- mac_processor:
  Intel
- md5:
  04c1004fa2b0de119d764c4ec41e567d
- military_apo:
  PSC 0750, Box 3490
- military_dpo:
  Unit 1372 Box 0808
- military_ship:
  USNV
- military_state:
  AE
- mime_type:
  message/rfc822
- month:
  11
- month_name:
  February
- name:
  Dorcas Wolff MD
- name_female:
  Amy Tromp
- name_male:
  Colvin Hoppe
- null_boolean:
  False
- numerify:
  597
- opera:
  Opera/9.27.(Windows NT 6.2; it-IT) Presto/2.9.173 Version/11.00
- paragraph:
  Ex est maiores et maiores id voluptatem velit iure. Ratione ad enim sed. Quidem ea vel esse quis molestiae quisquam tempora.
- paragraphs:
  [u'Ipsum consequatur et et et et. Voluptatem qui totam veritatis. Quasi vitae autem quos magnam ducimus. Doloremque minima perferendis quibusdam mollitia doloremque. Et temporibus porro quia eveniet sunt.', u'Neque nihil cupiditate sunt. Velit ducimus sint qui eaque nam.', u'Sequi labore laudantium expedita iusto aspernatur quibusdam ipsa hic. Et delectus numquam et fugit. Et quo explicabo quia voluptatem molestias blanditiis omnis. Mollitia facilis molestiae sunt aut eius quia.']
- parse:
- password:
  H$E4isuB__
- phone_number:
  322-309-6514
- postalcode:
  36510
- postalcode_plus4:
  31785-2317
- postcode:
  63240
- prefix:
  Dr.
- prefix_female:
  Miss
- prefix_male:
  Mr.
- profile:
  {'website': [u'http://feest.com/', u'http://runolfsson.info/', u'http://www.wiegand.com/', u'http://www.rennerzemlak.org/'], 'username': u'bogisich.forest', 'name': u'Lorenz Friesen III', 'blood_group': 'A+', 'residence': u'87593 Douglas Pike\nBeahanberg, NY 08971-4439', 'company': u'Ratke and Sons', 'address': u'3721 Rath Springs\nMillerbury, AK 65443-3366', 'birthdate': '1985-10-14', 'sex': 'M', 'job': 'Scientist, research (medical)', 'ssn': u'658-53-8410', 'current_location': (Decimal('-69.179215'), Decimal('73.080343')), 'mail': u'natalee03@yahoo.com'}
- provider:
- pybool:
  False
- pydecimal:
  93907633.3459
- pydict:
  {u'non': u'Sint at et ut ut.', u'voluptas': datetime(2011, 6, 30, 13, 28, 24), u'in': u'Consequatur modi ut.', u'temporibus': u'nberge@gmail.com', u'quam': 5946, u'quia': -50185653292693.0, u'corporis': 44989.34539971, u'aspernatur': datetime(2001, 5, 8, 5, 0, 4), u'id': u'Voluptas et.', u'consequatur': u'Sint voluptatum quo.'}
- pyfloat:
  5.427466
- pyint:
  5831
- pyiterable:
  [u'Praesentium atque.', 510, u'Voluptatem.', u'Aspernatur culpa.', -86872010066.1, Decimal('1.39045051271E+12'), u'Aut neque quod quis.', 774237663.8, u'Libero voluptates.']
- pylist:
  [u'Enim voluptatem.', u'In ut unde aut est.', 3744, u'Tenetur harum nam.', u'Amet nam quam.', Decimal('-2.15222088569E+13'), 56014295154.367]
- pyset:
  set([2144, u'Est voluptas rem.', datetime(1989, 2, 18, 3, 52, 13), u'Est nihil.', Decimal('-243.7196'), datetime(1983, 9, 28, 12, 52, 56), u'beverlee76@yahoo.com', Decimal('-3.04622164413E+12'), u'Natus ut atque.'])
- pystr:
  Ex non explicabo.
- pystruct:
- pytuple:
- random_digit:
  5
- random_digit_not_null:
  1
- random_digit_not_null_or_empty:
  
- random_digit_or_empty:
  9
- random_element:
  b
- random_int:
  3804
- random_letter:
  x
- random_number:
  981
- randomize_nb_elements:
  11
- rgb_color:
  63,125,116
- rgb_color_list:
- rgb_css_color:
  rgb(21,8,176)
- safari:
  Mozilla/5.0 (iPod; U; CPU iPhone OS 3_1 like Mac OS X; en-US) AppleWebKit/533.9.6 (KHTML, like Gecko) Version/3.0.5 Mobile/8B112 Safari/6533.9.6
- safe_color_name:
  fuchsia
- safe_email:
  irvin.little@example.net
- safe_hex_color:
  #dd7700
- secondary_address:
  Suite 378
- seed:
  None
- sentence:
  Consequuntur unde quo non voluptas minima voluptatibus.
- sentences:
  [u'Sapiente quia incidunt porro dolorum sapiente.', u'Et quaerat ad nesciunt maiores.', u'Ut excepturi autem qui laboriosam.']
- set_formatter:
- sha1:
  aab5c833e76265e48aa35b5b39ed798b2d21d971
- sha256:
  d74cbb45b74394c6a95f55f27926db1bdfeaf42dbbfcb3fddff22911f0dd4202
- simple_profile:
  {'username': u'cbuckridge', 'name': u'Britta Cartwright', 'birthdate': '1974-08-05', 'sex': 'M', 'address': u'123 Talia Island\nSouth Thursa, DC 68141', 'mail': u'vonda67@hotmail.com'}
- slug:
  ut-ducimus-illum
- ssn:
  142-04-9440
- state:
  Vermont
- state_abbr:
  VT
- street_address:
  33334 Auer Via
- street_name:
  Tayler Pines
- street_suffix:
  Curve
- suffix:
  II
- suffix_female:
  DVM
- suffix_male:
  V
- text:
  Eius ut atque voluptatum deserunt similique aliquid. Rerum quo commodi cum eius.
- time:
  14:45:00
- time_delta:
  16412 days, 22:34:37
- timezone:
  Asia/Saigon
- tld:
  com
- unix_time:
  330977642
- uri:
  http://boyer.com/main/index.php
- uri_extension:
  .html
- uri_page:
  post
- uri_path:
  tags/posts
- url:
  http://www.hammes.com/
- user_agent:
  Mozilla/5.0 (compatible; MSIE 7.0; Windows 98; Trident/4.0)
- user_name:
  jenkins.layton
- uuid4:
  8fef0d23-2010-44f1-873a-49767169db27
- windows_platform_token:
  Windows 95
- word:
  omnis
- words:
  [u'neque', u'odit', u'dolorum']
- year:
  1975
- zipcode:
  77130
- zipcode_plus4:
  71629-5918

Set up logging with Flask and gunicorn

Let's say that we want to be able to log in the specified log file and syslog. Our application - is a simple app.py file. The first step, I suggest, is to set logger name:

import os

from flask import Flask


app = Flask(__name__)
app.logger_name = "flask.app"

# ...

if __name__ == "__main__":
    app.run()

You can log in your project like this:

app.logger.info("Hello")

A log configuration file for the gunicorn may look something like this:

[loggers]
keys=root, gunicorn.error, gunicorn.access, requests.packages.urllib3.connectionpool, __main__

[handlers]
keys=log_file, syslog

[formatters]
keys=generic

[logger_root]
level=INFO
handlers=log_file, syslog

[logger___main__]
level=DEBUG
handlers=log_file, syslog
propagate=0
qualname=__main__

[logger_gunicorn.error]
level=INFO
handlers=log_file, syslog
propagate=0
qualname=gunicorn.error

[logger_gunicorn.access]
level=INFO
handlers=log_file, syslog
propagate=0
qualname=gunicorn.access

[logger_requests.packages.urllib3.connectionpool]
level=WARN
handlers=log_file, syslog
propagate=0
qualname=requests.packages.urllib3.connectionpool

[handler_syslog]
class=logging.handlers.SysLogHandler
formatter=generic
args=()

[handler_log_file]
class=logging.FileHandler
formatter=generic
args=('/home/myuser/log/gunicorn.log',)

[formatter_generic]
format=%(asctime)s [%(process)d:%(name)s:%(lineno)s] [%(levelname)s] %(message)s
datefmt=%Y-%m-%d %H:%M:%S
class=logging.Formatter

The last step is starting gunicorn:

gunicorn app:app --log-config /home/myuser/gunicorn_logging.conf

bash: How to remove all *.pyc files

Remove all *.pyc files from the current directory (and subdirectories):

find . -name \*.pyc -delete

How to find outdated python packages in your environment

The easiest way to find out if updated version of any installed package is available is to run that command:

pip list --outdated

There is also a special package for that task which is called pip-tools

python: Precompile nunjucks files on change

Preface

I have been writing an Firefox extension for some time. It generates HTML content which depends on the context. When I started, I decided to do it with JavaScript:

var container = document.createElement("div");
document.getElementById("items-list").appendChild(container);

However, JavaScript quickly became messy. Another problem was that I didn't have HTML in front of me to create stylesheet.

I have used JavaScript template engine in one of my projects recently. It would be nice to do it in the Firefox extension. There will be at least two benefits: my JavaScript will be shorter and cleaner to read and I will be able to see HTML to easy write css.

I choose nunjucks template engine by Mozilla. It is fast, feature rich and copies Jinja2 syntax. To be honest, the last reason was the most important for me. I have developed several web applications in Django and Flask and really like django templates and jinja2 syntax.

Nunjucks easy integrates in the webpage. It is as simple as that:
1. Load script in you HTML:

<script src="nunjucks.js"></script>

2. Render template with context and append it to the page:

var rendered_html = nunjucks.render('templates/index.html', { foo: 'bar' });
document.getElementById("container").innerHTML = rendered_html;

I tried the same approach in the Firefox extension. I needed to append HTML in the toggle button popup container. So I loaded nunjucks.js inside popup (it is called panel in the Firefox) by passing it to contentScriptFile option.

Problem

Unfortunately, nunjucks wasn't able to find template I wanted it to render. As far as I know It makes Ajax request to your server (what server in case of Firefox extension, ha?) to retrieve HTML file and then renders it. Nunjucks also supports precompiled templates (which indeed it renders much faster). Precompiled templates are preferable in the production and are represented as JavaScript files. So my idea was to precompile templates and pass these JavaScript files with contentScriptFile option. In the nunjucks.render function it looks for the array instance with the same name as path to the Ajax request('templates/index.html' in the example above) and if it exists it starts to render it without downloading an html template. Exactly what we need!

Templates are compiled into JavaScript files by precompile bash script. It is inside bin folder of the nunjucks git repository. Lets clone the repository:

cd /home/jsn/app/git_projects/
git clone https://github.com/mozilla/nunjucks.git

We also need to install some dependencies. Firstly, we install nodejs package and nodejs package manager:

sudo yum install nodejs npm

Then two nodejs packages:

npm install chokidar optimist

No we are able to compile template into JavaScript file with command:

/home/jsn/app/git_projects/nunjucks/bin/precompile path_to_html >> path_to_js

There is a problem with that command: it looks ugly because it demands absolute paths to the HTML and JavaScript files. If you run it and open compiled file you will see that precompile script used absolute path_to_template to identify template. That is not what we need as we are going to use just filename as identifier ("index.html"). To fix last problem we can pass --name to the precompile command:

/home/jsn/app/git_projects/nunjucks/bin/precompile --name index.html path_to_the_index.html >> path_to_the_index.js

Now we have the command which we are supposed to execute on every HTML file change. It would be nice to automate that step. So we need a script which will check every template in our templates folder and if a file was changed it would compile the file.

Solution

We are going to write that script in Python. First of all we need list of all files inside template folder. We will use listdir method of os package:

file_list = os.listdir(templates_dir)

File modification time we can check with os.getmtime() method. We will create global dictionary FILES. The keys of that dictionary will be file names and values will be last modification time. All we need to do is to fill that dictionary on the script start up and run precompile command if last modification time was changed.

FILES = {}
TIMER = 1

COLOUR_END = '\033[0m'
COLOUR_GREEN = '\033[92m'
COLOUR_BLUE = '\033[94m'


def check_templates(templates_dir=TEMPLATES_DIR):
    file_list = os.listdir(templates_dir)
    for item in file_list:
        if item.split(".")[-1] == "html":
            modified = os.path.getmtime(templates_dir + "/" + item)
            if item in FILES:
                if FILES[item] != modified:
                    FILES[item] = modified
                    precompile(templates_dir, item)
            else:
                print (COLOUR_BLUE + item + " is being watched" + COLOUR_END)
                FILES[item] = modified
                precompile(templates_dir, item)

We also filter files in our template directory by extension (as we are interested only in the html files). I use print function to inform myself when script finds new file in directory. Note, that these lines will be blue. It will make that event more noticeable.

We will check files for changes every second (TIMER variable):

if __name__ == '__main__':
    while True:
        check_templates()
        sleep(TIMER)

We can ask Python to execute for us any command we would normally execute in bash. Call method of the subprocess package helps to do it.

def precompile(tempates_dir, filename):
    compiled_filename = filename.replace("html", "js")
    path_to_js = tempates_dir + "/" + compiled_filename
    path_to_html = tempates_dir + "/" + filename
    if os.path.exists(path_to_js):
        call("rm -f " + path_to_js, shell=True)
    command = NUNJUCKS_REPO + "/bin/precompile --name " + filename + " " + path_to_html
    command = command + " >> " + path_to_js
    call(command, shell=True)
    print(COLOUR_GREEN + datetime.now().strftime("%X" + " " + filename + " ...OK") + COLOUR_END)

By default if compiled JavaScript exists, precompile command will add newly compiled data to it, so we have to check if compiled file exists with os.path.exists() method and remove it with bash rm -f command. NUNJUCKS_REPO is the path to the cloned nunjucks repository.

There is still a thing we can improve in the script. I will store that script in the project repository and I would like to use it on different machines. So hard coded template and nunjucks directory paths is not the option. I would like to set them as commands arguments to our script or maybe I will set these paths as environment variables. Anyway, full version of the script is below:

#!/usr/bin/python

# This script will monitor templates folder and automatically compile nunjucks html templates on change

import os
import sys
from datetime import datetime
from subprocess import call
from time import sleep

try:
    NUNJUCKS_REPO = sys.argv[1]
except IndexError:
    NUNJUCKS_REPO = os.getenv("NUNJUCKS_REPO")
try:
    TEMPLATES_DIR = sys.argv[2]
except IndexError:
    TEMPLATES_DIR = os.getenv("TEMPLATES_DIR")

FILES = {}
TIMER = 1

COLOUR_END = '\033[0m'
COLOUR_GREEN = '\033[92m'
COLOUR_BLUE = '\033[94m'


def check_templates(templates_dir=TEMPLATES_DIR):
    file_list = os.listdir(templates_dir)
    for item in file_list:
        if item.split(".")[-1] == "html":
            modified = os.path.getmtime(templates_dir + "/" + item)
            if item in FILES:
                if FILES[item] != modified:
                    FILES[item] = modified
                    precompile(templates_dir, item)
            else:
                print (COLOUR_BLUE + item + " is being watched" + COLOUR_END)
                FILES[item] = modified
                precompile(templates_dir, item)


def precompile(tempates_dir, filename):
    compiled_filename = filename.replace("html", "js")
    path_to_js = tempates_dir + "/" + compiled_filename
    path_to_html = tempates_dir + "/" + filename
    if os.path.exists(path_to_js):
        call("rm -f " + path_to_js, shell=True)
    command = NUNJUCKS_REPO + "/bin/precompile --name " + filename + " " + path_to_html
    command = command + " >> " + path_to_js
    call(command, shell=True)
    print(COLOUR_GREEN + datetime.now().strftime("%X" + " " + filename + " ...OK") + COLOUR_END)


if __name__ == '__main__':
    while True:
        check_templates()
        sleep(TIMER)

As we use only precompiled JavaScript templates we can switch to nunjucks-slim.js. Slim version is smaller and works only with precompiled templates.

If you have any suggestions or questions, please let me know in the comments.

python: Как отобразить прогресс задачи

В python есть очень хорошая встроенная библиотека для отображения прогресса задачи. Называется она progressbar. Установим ее с помощью pip:

sudo pip install progressbar

Для симуляции задачи будем использовать модуль time.

import progressbar
import time

pbar = progressbar.ProgressBar(widgets=[progressbar.Bar(), progressbar.Percentage(), ' ', progressbar.ETA()])
for i in pbar(range(50)):
time.sleep(0.2)

Вот так все просто. Наслаждаемся результатом:

После выполнения задачи будет показано время, затраченное на нее.